"cargo install" apparently ignores "Cargo.lock" as opposed to "cargo build"

[df5602]

I've encountered a strange issue today.

Background: The percent-encoding crate recently made a new major release. Knowing that stuff could break I held off on updating the crate because I haven't yet had time to check what changed.

Problem
When I compile my binary crate using cargo build (with and without --release) everything builds fine. However, if I compile my binary using cargo install --path . -f, compilation fails with an error related to percent-encoding (the breaking change warranting the major version bump I assume..). In the output I see that it compiled percent-encoding v2.0.0 instead of percent-encoding v1.0.1 as specified in Cargo.lock. (Note: The Cargo.toml doesn't specify a specific version.)

The crate is the following, if you want to try at home:
https://github.com/df5602/bingers

Steps

1. $ cargo clean
2. $ cargo build

...
Compiling percent-encoding v1.0.1
...
Finished dev [unoptimized + debuginfo] target(s) in 48.32s

3. $ cargo build --release

...
Compiling percent-encoding v1.0.1
...
Finished release [optimized] target(s) in 1m 25s

4. $ cargo install --path . -f

  Installing bingers v0.1.0 (/path/to/bingers)
    Updating crates.io index
   Compiling percent-encoding v2.0.0
   Compiling bingers v0.1.0 (/path/to/bingers)
error[E0432]: unresolved import `percent_encoding::QUERY_ENCODE_SET`
  --> src/tvmaze_api.rs:10:45
   |
10 | use percent_encoding::{utf8_percent_encode, QUERY_ENCODE_SET};
   |                                             ^^^^^^^^^^^^^^^^ no `QUERY_ENCODE_SET` in the root

error: aborting due to previous error

I can work around it by specifying an explicit version in Cargo.toml, but the observed behaviour was surprising to me...

Notes

Output of cargo version:

cargo 1.36.0 (c4fcfb725 2019-05-15)

OS: Mac

[ehuss]

This has been changed in 1.37 (beta). Starting in that version the --locked flag can be used to force cargo to use the Cargo.lock file.

The intent is that cargo install will behave the same whether it is from a registry or a local path.

We discussed this a bit (#6840 (review) is most relevant). Ideally Cargo would issue a message with a hint to try --locked, but I'm not so confident that would work well. I think since it can result in any error, it wouldn't be able to know if --locked would actually fix it, so I think it would be misleading too often.

[df5602]

1. I'm using cargo 1.36 (stable) which shouldn't show that behaviour yet, right? (Running cargo install with --locked worked, though...)

2. Reading up on the linked issue: if I understand correctly, going forward, the behaviour will be different between cargo build and cargo install regarding dependency resolution (at least for binary crates)? If yes, I would consider that as very surprising! (So, what's the Cargo.lock for? Basically like this, you compile, run your tests and then cargo install possibly picks different dependencies?)

I know, I should probably fix my Cargo.toml and specify proper versions instead of using "*". But until now, it never really mattered for binary crates. Updating dependencies was always a manual process, so that you can deliberately audit the changes if you wish to do so (for me it was basically running cargo outdated, check the listed crates and look for the changelog, and eventually running cargo update -p xyz).
If that behaviour changes, I would expect at least a warning à la "Warning: Ignoring lock file!" and probably a PSA in the Rust release notes.

[ehuss]

1. Running `cargo install` with `--locked` worked, though...

Ah, yea, in that sense there is no change in behavior. I mis-remembered how it used to work. The present stable behavior for install --path is that Cargo.lock is ignored by default, and --locked can be used to honor it. That hasn't changed. (What did change is installing from a registry, which didn't support lock files at all.)

It's difficult to balance the defaults here. I think it would be confusing if cargo used Cargo.lock differently based on the source. And for now we don't want to default to using Cargo.lock for installing from a registry. I think that's up for debate on changing it, but it was the conservative choice we made for now. If we ever decide to change it, we would probably want to add a flag to override it, and that adds more complexity.

[df5602]

Ah ok, so I guess it was the first time I ran into this because I deliberately held off on updating a dependency (due to breaking changes) which happens quite rarely...

It's good to know that this is the default behaviour, but I think this should be a lot better documented, because it's highly unintuitive and surprising:
Until now, my mental model of cargo install --path was basically "run cargo build --release and copy the binary somewhere". But apparently that's wrong, it should be "compile the binary slightly differently than cargo build --release and copy the binary somewhere".

I think it's the wrong default for binary crates because it means that you're deploying different binaries than you're testing! (In my opinion it would be ok, if cargo build would ignore lockfiles by default and you could explicitly opt-in to using lockfiles. But currently the default is otherwise.)

[ehuss]

I think this should be a lot better documented

There's a fairly lengthy paragraph on https://doc.rust-lang.org/nightly/cargo/commands/cargo-install.html that explains how cargo install handles Cargo.lock (this is new for 1.37, so it hasn't hit stable, yet). If there's any part that could be clarified let us know, we're always eager to improve the documentation.

[df5602]

I assume this will also be part of the cargo install --help page? I think the behaviour is described very clearly in the updated help page.

Given that the described behaviour is apparently expected behaviour, I think we can close this issue?

[MaulingMonkey]

https://doc.rust-lang.org/nightly/cargo/commands/cargo-install.html has some misleading documentation under "Manifest Options" for --frozen / --locked:

The --frozen flag also prevents Cargo from attempting to access the network to determine if it is out-of-date.

This is untrue for cargo install. In fact, I don't think this can ever succeed for install unless the crate has no dependencies. Maybe not even then.

These may be used in environments where you want to assert that the Cargo.lock file is up-to-date (such as a CI build) or want to avoid network access.

This is inaccurate for cargo install. If anything, you'd use these flags to use an "out-of-date" lock file - in the sense that all the documentation suggesting cargo install xyz will be using different versions.

If the lock file is missing, or it needs to be updated ...

This is technically accurate, but the way it implies it might not need to be updated is kinda confusing, as which crates are used will always be updated without these flags - for the install build at least - regardless of if they "need to be" or not.

If this documentation is automatically shared with other commands, it might at least be worth breifly mentioning cargo install being a special case. Maybe something like:

Unlike cargo build, cargo install will ignore Cargo.lock files completely without --locked."

?

Ideally Cargo would issue a message with a hint to try --locked, but I'm not so confident that would work well.

Instead of trying to guess when an error can be fixed by --locked, it might be useful to instead warn if Cargo.lock is present, and specifies different versions than are being used, since that's just begging for version confusion and building everything twice needlessly. --locked is one possible fix, but cargo update is another - since presumably most docs suggest cargo install xyz rather than cargo install xyz --locked?

[MaulingMonkey]

(I also suspect I landed on the "Manifest Options" section by searching the cargo install docs page for "--frozen", which caused me to skip over the earlier/better explaination of locking behavior entirely.)

[ehuss]

I assume this will also be part of the cargo install --help page?

Not for now, unfortunately we don't share the man pages with the help text. The long-term intent is to make it available in some fashion.

If this documentation is automatically shared

Yes, it is shared. We can probably make it special-cased for install, but that would take a bit of restructuring.

[df5602]

In that case I take back what I said earlier: I don't think I'd have the idea to look-up a description of a command online, if said command also has an extensive man page, so I still consider it undocumented behaviour...

I also agree with @MaulingMonkey: If this is the desired behaviour, then it should at least generate warnings, something along those lines (modulo wordsmithing):

Warning: Ignoring `Cargo.lock` file!
Detecting version mismatch:
    | Specified in `Cargo.lock` | Used in build
------------------------------------------------
foo |                    v1.0.1 |        v1.0.3

... possibly followed by a few suggestions on how to deal with the situation (i.e. use --locked, run cargo update, etc.)

(Although I don't know the inner workings of cargo, so I can't judge how much work that would be...)

[Nemo157]

(EDIT: Moved to #7495)

[matklad]

I've hit this in rust-analyzer a couple of times, and I would appreciate the warning. How does this interact with publish-lockfile? Will adding publish-lockfile = true to Cargo.toml make Cargo to use lockfile when installing? Or is the publish-lockfile feature effectively a no-op?