option to confirm expected package hashes when installing

[qwcode]

pip should offer some way to confirm expected hashes when installing
(mainly due to it being possible for the same pypi distribution to have a different hash over time; project authors are allowed to delete a distribution, and then upload a new one with the same name and version, but different hash.)

to be clear, pip already validates the hash url fragment in the PyPI link. this is not about doing that. this about confirming that you're getting the same hash that you've installed before, and want to keep getting.

peep offers this feature using requirement file comments: https://pypi.python.org/pypi/peep

as for implementing, here's one idea from @dstufft

"I keep playing with the idea of a lock file. e.g. in your requirements.txt you'd just specify your top level dependencies, then when you install it would resolve the dependencies, put them + hashes in requirements.txt.lock and next time you install it will look for a lockfile and install from there (implying --no-deps) so we don't get this thing where in order to do repeatable builds you basically have to manually pin every dependencies inside of a requirements file by hand"

cc @erikrose

[dstufft]

One problem to deal with is that you may (rightly) get two different files for the same project + version, for instance if there's a binary Wheel uploaded for Windows you might get that on Windows (or not if you don't use --use-wheel) and you might get something completely different for Linux.

I think peep just ignores this problem and says that a peep flavored requirements file is only promised to be valid on using the same install time options and the same target system.

[erikrose]

So, we have a couple of options:

1. A hole in the current requirements.txt grammar we can exploit is the "extras": pip install Paste[openid]. We could say that any "extra" with an equal sign in it is a key/value thing rather than an actual extra:

   Paste[sha256-win32=abcdefghjijklmnop, sha256-mac=zxcvbnmasdfghjkl]==1.2.3
   

   This would take care of peep hashes and any other future key/value needs. It wouldn't require distributors of requirements.txt files to change their grammars, and we wouldn't have to maintain two parallel parsers. It's a little ugly, but it's extensible, converting the hinky requirements.txt grammar into a tagged one without breaking compatibility with old pips.

2. The lockfile idea. I like this a lot better. It's elegant, decoupled, and backward-compatible. It's fail-safe iff we require an option to be passed in order to pay attention to the lockfile (actually, let's call it a hashfile so as not to overload a term). We can automatically lay down the hashes nondestructively without having to tiptoe around existing comments and formatting in a requirements.txt.
   I imagine most people wouldn't check requirement-hashes.txt files into source control; they would mostly be maintained by deployers. But in case they wanted to, support for multiple platforms would be trivial: have a separate hashfile for each platform: requirement-hashes-win32.txt, requirement-hashes-win-amd64.txt, etc. Or we could multiplex them into one file if we felt strongly enough and felt like writing more parsing code—probably not worth it, IMO. Dealing with the wheel/non-wheel duality will take a little more thinking, but I don't think it's a killer.

I'm getting pretty close to starting to merge peep into pip. I'll probably take option number 2 unless somebody stops me. Any further thoughts? I'm happy to put forward a more concrete strawman proposal so we don't waste time upfront bikeshedding.

[erikrose]

One other horrible truth: currently, requirements.txt supports any pip version specifier, including things like >=. If people use those—a questionable idea anyway—it'll lead to a lot of spurious hash mismatches. I suppose the best thing to do is to intelligently mention that in the error message, but I wonder: does anyone else have philosophical thoughts about the use of non-== specifiers?

[pnasrat]

I don't think we can break the large number of users who may be using >=. It is definitely in use

https://github.com/search?l=&p=2&q=%3E%3D++path%3Arequirements.txt&ref=advsearch&type=Code

It's probably more widely used in private repos too.

To be fair you are breaking http://pythonhosted.org/setuptools/pkg_resources.html#requirement-objects

Which EBNF is

requirement  ::= project_name versionspec? extras?
versionspec  ::= comparison version (',' comparison version)*
comparison   ::= '<' | '<=' | '!=' | '==' | '>=' | '>'
extras       ::= '[' extralist? ']'
extralist    ::= identifier (',' identifier)*
project_name ::= identifier
identifier   ::= [-A-Za-z0-9_]+
version      ::= [-A-Za-z0-9_.]+

So the extralist item should be an identifier and not contain = as you use above.

[erikrose]

Yes, obviously we would be changing the grammar. But we'd break old versions of pip only when used against versions of requirements.txt that actually include hashes. That's actually a desired behavior, as it fails safe.

But, like I said, I favor option 2 anyway.

[erikrose]

In any case, I certainly didn't mean to suggest dropping support for range specifiers. I was more curious about anyone had worked out a semantic boundary between using them in requirements files and using them in install_requires. I have a pretty good mental model of what == is for in requirements files—known-working sets or frozen deployment sets—but I don't have a good use case in mind when it comes to greater-than and less-than.

[dstufft]

I would prefer a lockfile (vs a hashfile) because Ideally it would do more than just hashes. I'm thinking the "normal' requirements.txt would no longer need to list every dependency and if the lockfile doesn't exist then pip will install from the requirements.txt, resolve the deps, write out a lockfile with a complete list of deps + hashes. If the lockfile does exist then it can just skip resolving deps and act like --no-deps was passed to it.

[dstufft]

But that may be more than you want to get into. It's something i've sketched out in my head a few times and just hadn't had time to work on yet.

[qwcode]

One other horrible truth: currently, requirements.txt supports any pip version specifier,
including things like >=. If people use those—a questionable idea anyway—it'll lead to a lot of spurious hash
mismatches

You're focused on the repeatability use case for requirements files.

Currently, requirements files are just a way to get in a lot of pip install args. That's how the pip code treats them.

There are 3 main purposes I think:

1. To achieve repeatability using == specifiers.
2. As a way to override what will happen just based on install_requires, e.g. to resolve conflicts (that pip's "first found wins" resolver logic doesn't properly do) Many specifier patterns besides == can make sense for this.
3. Saves typing and easier to reuse.

[qwcode]

I'm thinking the "normal' requirements.txt would no longer need to list every dependency

hmm, the phrase "no longer need" here concerns me.

whether it's a requirements file, or a "lock file", when repeatability is concerned, many people will want an overt user artifact that is managed in version control. Not sure of your intention, but the lock file can't just be a run-time by-product, that isn't specifically referenced.

and if the lockfile doesn't exist then pip will install from the requirements.txt, resolve the deps,

the lockfile would need to be something that can be used explictly, not just used if it exists.
otherwise, you can't ensure a repeatable process.

write out a lockfile with a complete list of deps + hashes.

btw, we can't assume write access next to the requirements file.
lock file creation would have to be an explicit command or option to some other command.

I'm concerned about growing a new "lock file" concept in addition to requirements files. We'll end up with three notions: install_requires, requirements files, and lock files. I'd thinking maybe just "requirements 2.0". Just a better requirements file format that supports hashes. Conceptually the same, just better.

[dstufft]

I think even if we have requirements 2.0 (which is also something i'd love to do) I think we want a lockfile. Bundler has this and it's really nice.

In order to have a repeatable process right now you have to manually manage the entire dependency graph inside of the requirements file. Pinning every single dependency and managing updating them or removing them when no longer needed. A lock file means that you only need to declare what your project depends on and pip itself manages pinning the entire dependency graph.

In Bundler land you have Gemfile instead of requirements.txt and the lockfile is called Gemfile.lock, if Gemfile.lock exists it uses that otherwise it resolves the dependencies and creates Gemfile.lock and then installs it. If you want to force the use of a .lock you run bundle install --deployment instead of just bundle install.

[qwcode]

manually manage the entire dependency graph inside of the requirements file.
Pinning every single dependency and managing updating them or removing them when no longer needed.

manual? we have pip freeze to help with that.

In Bundler land you have Gemfile instead of requirements.txt and the lockfile is called Gemfile.lock

hmm, I would say gemfile is like our install_requires', and lock files are like pip freeze-generated requirements files. so, we have "lock files" now. that just don't handle hashes. I'm more concerned with filling in that deficit, not a major conceptual makeover, which is what this would be.

[dstufft]

manual? we have pip freeze to help with that.

pip freeze helps but there's still a bunch of manual busy work to doing it. It's annoying to keep updated.

hmm, I would say gemfile is like our install_requires', and lock files are like pip freeze-generated requirements files.

No, install_requires would be gemspec. install_requires, requirements.txt, and the proposed lockfile do not include the same information (nor should they).

I'm more concerned with filling in that deficit, not a major conceptual makeover, which is what this would be.

I'm -1 on including another file for just the hashes. Either we should go all out and include a real lock file or we should figure out how to get the hashes into requirements.txt

[dstufft]

To be specific.

• install_requires - Holds "abstract" dependencies of a singular installable item. We want these to be as wide open as possible to prevent dependency hell.
• requirements.txt - User facing file, users don't want to manually manage every single dependency and figure out who still depends on it (if anyone etc). Often times these can/should be ranges. (>=1.2,<1.3)
• lockfile - Not user facing, has the exact version (==) pinned always. Already has the entire dependency graph solved so --no-deps can be assumed. Lots faster and easy to include things like hashes.

Now it's true you can approximate a lockfile with requirements.txt but that's not a very user friendly UX. At best case they'll want two files anyways and they'll need to delete a virtualenv, install everything, then pip freeze to their manually managed lockfile. At worst case they are going to be manually resolving the dependency graph inside of a single requirements.txt which is going to end up resolved wrong because humans are error prone.