Add a pip-style `--upgrade-strategy` setting to `pipenv lock`?

[ncoghlan]

pip install these days offers an --upgrade-strategy option to let users choose between "only-if-needed" upgrades (which upgrades components only if needed to satisfy new dependencies) and "eager" upgrades, which upgrades everything which has a newer version available that still meets the dependency constraints.

In pip 10, the default strategy is changing from eager to only-if-needed: pypa/pip#4500

pipenv lock currently follows the original pip install --upgrade policy of "upgrade everything".

This is actually fine for my own current use cases - most of my dependencies are stable enough that the combination of loose dependencies in Pipfile and eager upgrades in pipenv lock works well. I also think this remains the right default behaviour for pipenv.

However, I'm wondering if permitting pipenv lock --upgrade-strategy=only-if-needed may make pipenv applicable to more use cases.

[nateprewitt]

Thanks for bringing this to our attention, @ncoghlan! Do you know offhand what version of pip introduced --upgrade-strategy? My only concern adding this kind of functionality is that we may encounter conflicts with older versions of pip.

Our recent hiccup in Requests (psf/requests#4006) has me a bit paranoid about features that didn't exist back to pip 6.X, if not earlier. There are still at least 1.5 million people/instances using the 1.5.X series of pip, likely due to it coming from distribution repositories. I guess we could consider forcing a minimum version of pip though.

@kennethreitz, do you have any thoughts on supporting this. I do think we should maintain eager as the default after pip 10 is released.

[ncoghlan]

@nateprewitt It doesn't currently seem to be mentioned in the release notes [1], but pypa/pip#3972 is the PR that added it. That means it would have first appeared in pip 9.0.0 in November 2016.

[1] I submitted an issue regarding that oversight: pypa/pip#4568

[ncoghlan]

As far as the version compatibility question goes, I think it would be reasonable to have using --upgrade-strategy at the pipenv level simply fail if the underlying pip was too old (ideally with a nice error message explaining that at least pip 9.0.0 is needed, rather than letting the raw unknown argument error escape).

However, I also think you're right that the more important point here is to have pipenv pass --upgrade-strategy=eager on pip 10+ so that pipenv retains its current default behaviour. Allowing pipenv users to optionally pass --upgrade-strategy=only-if-needed would then just be a bonus.

[ncoghlan]

Adding @dstufft to the discussions, as he mentioned hoping to deprecate and remove the --upgrade-strategy option some day, and I think pipenv is an example of a use case where it makes sense to keep the old eager upgrade behaviour available indefinitely (just on an opt-in basis).

[stoggi]

This causes a problem for me. My Pipfile.lock specifies a specific version

"s3transfer": { "version": "==0.1.10"},

But when I run pipenv install a different version of s3transfer is installed 0.0.1. After a bit of investigation I figured out that pip was installing the correct version of s3transfer==0.1.10 but a subsequent package awscli was installing s3transfer==0.0.1 because it was specified as one of it's dependencies.

So, since the Pipfile.lock contains all dependencies of all your packages anyway, when running pip install from a Pipfile.lock we could be using the --no-deps option so that pip doesn't overwrite the dependencies at all.

[kennethreitz]

I definitely want to avoid adding options if at all possible, and your statement of "This is actually fine for my own current use cases" is a telling one, in this instance :)

[nateprewitt]

@stoggi, the issue you're experiencing is actually unrelated to this. The issue tracking your problem is #298. We'll need to have some work done on how we both validate Pipfiles and handle dependency conflicts. Neither pip or pipenv handle this well currently but hopefully will in a future release.

[ncoghlan]

I'm going to close this one, as I think the pip-tools model is going to be a better fit for the "only-if-needed" case, and I'm fine with folks having to drop down to that lower level tooling if they want more control over how their requirements are managed.

However, the change in pip's default upgrade strategy still needs to be accounted for at the pipenv level, so I filed #438 to cover that.

[brettdh]

I'd like to push back here. only-if-needed is a much more sensible default in my view, particularly coming from other package managers such as npm, yarn, and gem. If packages are automatically upgraded when doing so is not required, then the lockfile is not a lockfile; it's just a suggestion.

The whole point of the lockfile is to pin dependencies to a known working version - to not have to assume that "most of my dependencies are stable enough"; to make all upgrades explicit (which, of course, is better than implicit).

I started this line of conversation in #966, but maybe this (or another issue) is a better place for it. Having --upgrade-strategy in pipenv would be a fine stopgap, but I maintain that only-if-needed is a more sensible default, as evidenced by pip 10 and the aforementioned package managers. The current situation doesn't allow either, which makes pipenv a less reliable tool for locking down dependencies.