Validate VCS urls in hash-checking mode using their commit hashes

[jcushman]

What's the problem this feature will solve?

It is often useful to temporarily switch a requirement to a VCS URL, for example to refer to a pull request in a fork prior to release, and then switch back once the feature is released.

This practice is incompatible with --require-hashes, however. If any requirements are hashed, then the VCS URL requirement is rejected with "The editable requirement cannot be installed when requiring hashes, because there is no single file to hash" or "Can't verify hashes for these requirements because we don't have a way to hash version control repositories."

This makes it more difficult to use hashes, which ultimately discourages secure pip use.

Describe the solution you'd like

Many VCS URLs contain sha1 hashes in the refspec. (sha1 is not a great hash -- more discussion below.) A requirement like git+https://github.com/requests/requests.git@e52932c427438c30c3600a690fb8093a1d643ef3#egg=requests could be accepted with --require-hashes as long as the commit validates.

Support on the pip side would help to encourage use of hashes by pip-tools users. A user could include git+https://github.com/requests/requests.git@master#egg in requirements.in, and use pip-compile --generate-hashes to write a pinned VCS URL along with other hashed requirements to requirements.txt. Currently pip-compile --generate-hashes necessarily generates an uninstallable requirements.txt with VCS URLs, which discourages use of --generate-hashes.

Alternative Solutions

• Don't require hashes for VCS URLs, on the grounds that some hashes are better than none. (I believe this is the approach taken by pipenv, though I haven't verified.) This still checks hashes for most requirements, and the security risk of allowing some unhashed URLs is "attacker has compromised communications with a specific VCS server", which for many users would not significantly change their risk profile.

• (Provide option to disable hash-checking #4344) Same as above, but as an opt-in flag, either with a global flag or a --hash=skip flag per line.

• Keep VCS URLs in a separate requirements file, as recommended on editable cannot be installed when requiring hashes #4995. This adds complexity to the install process without any additional security over the flag option.

• Some VCS URLs can be converted to artifact URLs, like https://github.com/requests/requests/archive/e52932c427438c30c3600a690fb8093a1d643ef3.zip#egg=requests, which allows them to be hashed. This doesn't work for other packages, however, such as those that use setuptools_scm to set their version. For example, pip install https://github.com/jazzband/pip-tools/archive/f97e62ecb0d9b70965c8eff952c001d8e2722e94.zip will fail with "setuptools-scm was unable to detect version".

Additional context

This feature request was discussed when --require-hashes was first added and ultimately rejected by the author because of the insecurity of sha1:

@jezdez on Oct 8, 2015 Contributor
I was wondering about that, wouldn't some VCSs at least allow providing a hash for a repo? Is that out of scope and need to be added at some point? Or just a silly idea?

@erikrose on Oct 8, 2015 Author Contributor
So yes, this is something I thought we might add in the future: pip would recognize hash-based git refspecs as okay for hash-checking mode. It would run git fsck over them to make sure the hashes really match. (git doesn't check them implicitly, at least in older versions, allowing a malicious git server to pass off whatever it wants.) My only reservation is that SHA1 isn't considered a very collision-resistant hash anymore.

@sigmavirus24 on Oct 12, 2015 Member
git fsck is good except that sometimes genuine commits are bad. requests has a commit that was accepted through a PR and which has (much later) been determined to be invalid by git fsck. There's nothing we can do now without rewriting all of the history since that commit if we were to fix it.

@erikrose
erikrose on Oct 12, 2015 Author Contributor
I assume later commits fsck fine, correct? I'd be willing to accept that broken commits can't be used as hash-checked VCS checkouts by pip. But the point is moot because SHA1 will probably be cost-effectively breakable in a few years.

Since then, there's been a lot more discussion of the risks of using sha1 for refspecs following the ShAttered attack (where Google paid to create a PDF hash collision, and could have done the same for a git commit).

After that attack:

• Github guarded against sha1 collisions
• git hardened its sha1 implementation and is implementing sha256
• Mercurial added a page discussing limitations on the attack.

In short, a sha1 collision attack requires attackers to first issue a specially-crafted, suspicious benign commit, at great expense, and then replace it with a malicious one. So a security model that is concerned with sha1 attacks assumes that an attacker has already crafted a seemingly-benign commit and had it distributed -- by which point, the Mercurial page argues, there are cheaper, easier, and more deniable ways to succeed in the attacker's aim.

That argument is certainly debatable! So I'm hoping to discuss it here: would it be worth counting sha1 refspecs as "hashed" for now, in order to encourage use of pip install --require-hashes and pip-compile --generate-hashes?

[erikrose]

I think you (and others) make a good case that stonewalling SHA1 VCS URLs as unhashable is counterproductive in a lot of practical cases, like when using secure internal git servers. The obvious options I see are…

• Just letting them through. Not really my favorite, as it downgrades security silently. But maybe no users care.
• Adding an --almost-require-hashes option or similar. Seems okay on the face of it.

However, I will put this here for everyone's evaluation. Seems like SHA1 collisions are about to get another notch cheaper: https://www.zdnet.com/article/sha-1-collision-attacks-are-now-actually-practical-and-a-looming-danger/.

In my ideal world, git would release SHA256 support, and we'd have no reservations at all.

[jcushman]

@erikrose:

Adding an --almost-require-hashes option or similar. Seems okay on the face of it.

How about a special --hash value, like:

# accept the hash value provided in the VCS url:
git+https://github.com/requests/requests.git@e52932c427438c30c3600a690fb8093a1d643ef3#egg=requests --hash=vcs

or

# perform no hash validation on this line:
git+https://localhost:9000/requests/requests.git@e52932c427438c30c3600a690fb8093a1d643ef3#egg=requests --hash=skip

[erikrose]

That also occurred to me, but I thought it'd be best to make it clear to the runner of pip rather than just the author of the requirements file. But, now that you mention it, a package author doesn't (ideally) include hashes, as that would frustrate downstream integrators. So it's fine.

[funtikas25]

hello pleas help what the hash is hier ???????
[18:21:27] [WARNING] there has been a problem while writing to the session file ('OperationalError: attempt to write a readonly database')
[18:21:30] [INFO] retrieved: 07b1e0439ba077bc7bdfbbf2341b1d1f:dc
[18:21:30] [DEBUG] performed 249 queries in 380.19 seconds
[18:21:30] [INFO] retrieving the length of query output
[18:21:30] [INFO] retrieved: 21
[18:23:18] [DEBUG] performed 10 queries in 107.48 seconds
[18:23:18] [DEBUG] starting 8 threads
[18:24:14] [INFO] retrieved: o_i________________ 2/21 (9%)

[awbirdsall]

I've run into this same problem. I just want to point out another limitation with using an artifact url like https://github.com/requests/requests/archive/e52932c427438c30c3600a690fb8093a1d643ef3.zip#egg=requests is, as far as I can tell, there isn't good support by pip/pip-tools for working with this style url for a private repo. (jazzband/pip-tools#947 (comment))

[sbidoul]

I would be in favor of having --require-hashes accept such VCS URLs too.

In #6851 I added is_immutable_rev_checkout that could help implementing this.

[lcarva]

Why not just hash the whole VCS directory and use whatever hash algorithm the user wants?

This would probably have to exclude things like the .git directory which may(?) contain different files depending on the version of git used.