Update to the 2021 edition via `cargo fix` breaks code

[weiznich]

I tried to update a project with the following dependencies:

[dependencies]
diesel = {version = "1.4.7", features = ["postgres"]}
diesel_migrations = "1.4.0"

(Just having an empty project with those dependencies sufficient to reproduce the issue)

I executed the following commands as outlined here:

$ cargo +beta check # to verify that the project builds successful on the 2018 edition
$  cargo +beta fix --edition
$ sed 's/edition = "2018"/edition = "2021"/g' Cargo.toml -i
$ cargo +beta check # to check the projects builds successful on the 2021 edition

I expected to see this happen: Everything succeeds without errors

Instead, this happened: The final check commando fails with error messages

$ cargo +beta fix --edition                                                             
note: Switching to Edition 2021 will enable the use of the version 2 feature resolver in Cargo.
This may cause some dependencies to be built with fewer features enabled than previously.
More information about the resolver changes may be found at https://doc.rust-lang.org/nightly/edition-guide/rust-2021/default-cargo-resolver.html
When building the following dependencies, the given features will no longer be used:

  diesel v1.4.7 (as host dependency) removed features: 32-column-tables, bitflags, default, postgres, pq-sys, with-deprecated

note: This project appears to use both diesel and diesel_migrations. These packages have
a known issue where the build may fail due to the version 2 resolver preventing
feature unification between those two packages. See
<https://github.com/rust-lang/cargo/issues/9450> for some potential workarounds.

    Checking diesel-test-2021 v0.1.0 (/tmp/diesel-test-2021)
   Migrating src/main.rs from 2018 edition to 2021
    Finished dev [unoptimized + debuginfo] target(s) in 0.31s
$ sed 's/edition = "2018"/edition = "2021"/g' Cargo.toml -i                  
$ cargo +beta check      
   Compiling diesel v1.4.7
error[E0432]: unresolved import `diesel`
  --> /home/weiznich/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/diesel-1.4.7/src/sql_types/mod.rs:37:48
   |
37 | #[derive(Debug, Clone, Copy, Default, QueryId, SqlType)]
   |                                                ^^^^^^^ could not find `pg` in `diesel`
   |
   = note: this error originates in the derive macro `SqlType` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0432]: unresolved import `diesel`
  --> /home/weiznich/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/diesel-1.4.7/src/sql_types/mod.rs:75:48
   |
75 | #[derive(Debug, Clone, Copy, Default, QueryId, SqlType)]
   |                                                ^^^^^^^ could not find `pg` in `diesel`
   |
   = note: this error originates in the derive macro `SqlType` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0432]: unresolved import `diesel`
  --> /home/weiznich/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/diesel-1.4.7/src/sql_types/mod.rs:96:48
   |
96 | #[derive(Debug, Clone, Copy, Default, QueryId, SqlType)]
   |                                                ^^^^^^^ could not find `pg` in `diesel`
   |
   = note: this error originates in the derive macro `SqlType` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0432]: unresolved import `diesel`
   --> /home/weiznich/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/diesel-1.4.7/src/sql_types/mod.rs:115:48
    |
115 | #[derive(Debug, Clone, Copy, Default, QueryId, SqlType)]
    |                                                ^^^^^^^ could not find `pg` in `diesel`
    |
    = note: this error originates in the derive macro `SqlType` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0432]: unresolved import `diesel`
   --> /home/weiznich/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/diesel-1.4.7/src/sql_types/mod.rs:136:48
    |
136 | #[derive(Debug, Clone, Copy, Default, QueryId, SqlType)]
    |                                                ^^^^^^^ could not find `pg` in `diesel`
    |
    = note: this error originates in the derive macro `SqlType` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0432]: unresolved import `diesel`
   --> /home/weiznich/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/diesel-1.4.7/src/sql_types/mod.rs:155:48
    |
155 | #[derive(Debug, Clone, Copy, Default, QueryId, SqlType)]
    |                                                ^^^^^^^ could not find `pg` in `diesel`
    |
    = note: this error originates in the derive macro `SqlType` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0432]: unresolved import `diesel`
   --> /home/weiznich/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/diesel-1.4.7/src/sql_types/mod.rs:177:48
    |
177 | #[derive(Debug, Clone, Copy, Default, QueryId, SqlType)]
    |                                                ^^^^^^^ could not find `pg` in `diesel`
    |
    = note: this error originates in the derive macro `SqlType` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0432]: unresolved import `diesel`
   --> /home/weiznich/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/diesel-1.4.7/src/sql_types/mod.rs:205:48
    |
205 | #[derive(Debug, Clone, Copy, Default, QueryId, SqlType)]
    |                                                ^^^^^^^ could not find `pg` in `diesel`
    |
    = note: this error originates in the derive macro `SqlType` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0432]: unresolved import `diesel`
   --> /home/weiznich/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/diesel-1.4.7/src/sql_types/mod.rs:248:48
    |
248 | #[derive(Debug, Clone, Copy, Default, QueryId, SqlType)]
    |                                                ^^^^^^^ could not find `pg` in `diesel`
    |
    = note: this error originates in the derive macro `SqlType` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0432]: unresolved import `diesel`
   --> /home/weiznich/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/diesel-1.4.7/src/sql_types/mod.rs:278:48
    |
278 | #[derive(Debug, Clone, Copy, Default, QueryId, SqlType)]
    |                                                ^^^^^^^ could not find `pg` in `diesel`
    |
    = note: this error originates in the derive macro `SqlType` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0432]: unresolved import `diesel`
   --> /home/weiznich/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/diesel-1.4.7/src/sql_types/mod.rs:298:48
    |
298 | #[derive(Debug, Clone, Copy, Default, QueryId, SqlType)]
    |                                                ^^^^^^^ could not find `pg` in `diesel`
    |
    = note: this error originates in the derive macro `SqlType` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0432]: unresolved import `diesel`
   --> /home/weiznich/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/diesel-1.4.7/src/sql_types/mod.rs:313:48
    |
313 | #[derive(Debug, Clone, Copy, Default, QueryId, SqlType)]
    |                                                ^^^^^^^ could not find `pg` in `diesel`
    |
    = note: this error originates in the derive macro `SqlType` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0432]: unresolved import `diesel`
   --> /home/weiznich/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/diesel-1.4.7/src/sql_types/mod.rs:336:48
    |
336 | #[derive(Debug, Clone, Copy, Default, QueryId, SqlType)]
    |                                                ^^^^^^^ could not find `pg` in `diesel`
    |
    = note: this error originates in the derive macro `SqlType` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0433]: failed to resolve: could not find `pg` in `diesel`
  --> /home/weiznich/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/diesel-1.4.7/src/sql_types/mod.rs:37:48
   |
37 | #[derive(Debug, Clone, Copy, Default, QueryId, SqlType)]
   |                                                ^^^^^^^ could not find `pg` in `diesel`
   |
   = note: this error originates in the derive macro `SqlType` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0433]: failed to resolve: could not find `pg` in `diesel`
  --> /home/weiznich/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/diesel-1.4.7/src/sql_types/mod.rs:75:48
   |
75 | #[derive(Debug, Clone, Copy, Default, QueryId, SqlType)]
   |                                                ^^^^^^^ could not find `pg` in `diesel`
   |
   = note: this error originates in the derive macro `SqlType` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0433]: failed to resolve: could not find `pg` in `diesel`
  --> /home/weiznich/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/diesel-1.4.7/src/sql_types/mod.rs:96:48
   |
96 | #[derive(Debug, Clone, Copy, Default, QueryId, SqlType)]
   |                                                ^^^^^^^ could not find `pg` in `diesel`
   |
   = note: this error originates in the derive macro `SqlType` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0433]: failed to resolve: could not find `pg` in `diesel`
   --> /home/weiznich/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/diesel-1.4.7/src/sql_types/mod.rs:115:48
    |
115 | #[derive(Debug, Clone, Copy, Default, QueryId, SqlType)]
    |                                                ^^^^^^^ could not find `pg` in `diesel`
    |
    = note: this error originates in the derive macro `SqlType` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0433]: failed to resolve: could not find `pg` in `diesel`
   --> /home/weiznich/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/diesel-1.4.7/src/sql_types/mod.rs:136:48
    |
136 | #[derive(Debug, Clone, Copy, Default, QueryId, SqlType)]
    |                                                ^^^^^^^ could not find `pg` in `diesel`
    |
    = note: this error originates in the derive macro `SqlType` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0433]: failed to resolve: could not find `pg` in `diesel`
   --> /home/weiznich/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/diesel-1.4.7/src/sql_types/mod.rs:155:48
    |
155 | #[derive(Debug, Clone, Copy, Default, QueryId, SqlType)]
    |                                                ^^^^^^^ could not find `pg` in `diesel`
    |
    = note: this error originates in the derive macro `SqlType` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0433]: failed to resolve: could not find `pg` in `diesel`
   --> /home/weiznich/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/diesel-1.4.7/src/sql_types/mod.rs:177:48
    |
177 | #[derive(Debug, Clone, Copy, Default, QueryId, SqlType)]
    |                                                ^^^^^^^ could not find `pg` in `diesel`
    |
    = note: this error originates in the derive macro `SqlType` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0433]: failed to resolve: could not find `pg` in `diesel`
   --> /home/weiznich/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/diesel-1.4.7/src/sql_types/mod.rs:205:48
    |
205 | #[derive(Debug, Clone, Copy, Default, QueryId, SqlType)]
    |                                                ^^^^^^^ could not find `pg` in `diesel`
    |
    = note: this error originates in the derive macro `SqlType` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0433]: failed to resolve: could not find `pg` in `diesel`
   --> /home/weiznich/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/diesel-1.4.7/src/sql_types/mod.rs:248:48
    |
248 | #[derive(Debug, Clone, Copy, Default, QueryId, SqlType)]
    |                                                ^^^^^^^ could not find `pg` in `diesel`
    |
    = note: this error originates in the derive macro `SqlType` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0433]: failed to resolve: could not find `pg` in `diesel`
   --> /home/weiznich/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/diesel-1.4.7/src/sql_types/mod.rs:278:48
    |
278 | #[derive(Debug, Clone, Copy, Default, QueryId, SqlType)]
    |                                                ^^^^^^^ could not find `pg` in `diesel`
    |
    = note: this error originates in the derive macro `SqlType` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0433]: failed to resolve: could not find `pg` in `diesel`
   --> /home/weiznich/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/diesel-1.4.7/src/sql_types/mod.rs:298:48
    |
298 | #[derive(Debug, Clone, Copy, Default, QueryId, SqlType)]
    |                                                ^^^^^^^ could not find `pg` in `diesel`
    |
    = note: this error originates in the derive macro `SqlType` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0433]: failed to resolve: could not find `pg` in `diesel`
   --> /home/weiznich/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/diesel-1.4.7/src/sql_types/mod.rs:313:48
    |
313 | #[derive(Debug, Clone, Copy, Default, QueryId, SqlType)]
    |                                                ^^^^^^^ could not find `pg` in `diesel`
    |
    = note: this error originates in the derive macro `SqlType` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0433]: failed to resolve: could not find `pg` in `diesel`
   --> /home/weiznich/.cargo/registry/src/github.202132.xyz-1ecc6299db9ec823/diesel-1.4.7/src/sql_types/mod.rs:336:48
    |
336 | #[derive(Debug, Clone, Copy, Default, QueryId, SqlType)]
    |                                                ^^^^^^^ could not find `pg` in `diesel`
    |
    = note: this error originates in the derive macro `SqlType` (in Nightly builds, run with -Z macro-backtrace for more info)

Some errors have detailed explanations: E0432, E0433.
For more information about an error, try `rustc --explain E0432`.
note: This error may be due to an interaction between diesel and Cargo's new
feature resolver. Some workarounds you may want to consider:
- Add a build-dependency in Cargo.toml on diesel to force Cargo to add the appropriate
  features. This may look something like this:

    [build-dependencies]
    diesel = { version = "1.4.7", features = ["postgres"] }

- Try using the previous resolver by setting `resolver = "1"` in `Cargo.toml`
  (see <https://doc.rust-lang.org/cargo/reference/resolver.html#resolver-versions>
  for more information).

error: could not compile `diesel` due to 26 previous errors

The final error message already gives an indication why this error happens. (As already outlined here and here I feel that this change is nothing that is covered by any edition RFC, as any RFC states that changing the edition should be a crate local change and should not affect any upstream or down stream crates. That's at least in my opinion clearly not the case here. That written I accept that removing this feature from the 2021 edition will likely not happen so let's stop that discussion here 😞 )

To give a more productive indication on how that workflow can be improved:

• In an ideal world cargo fix --edition would just fix the problem and either add the corresponding section to Cargo.toml
• It would be possible that cargo fix --edition just sets resolver = "1" in the Cargo.toml" for edition upgrades and only for new projects the resolver = "2"feature is used (and that it is set explicitly in theCargo.toml` rather than just changing the default)
• As another show stop measurement cargo fix --edition should emit the same warning as cargo check already does. As the messaging currently reads it's unclear if the build will fail afterwards or not. Only cargo check gives the user a clear indication how to actually fix that problem
• As a medium term thing: Rust needs to provide better tools to bundle proc macro crates and other crates.

Meta

rustc --version --verbose:

rustc 1.56.0-beta.2 (f9a839ea0 2021-09-11)
binary: rustc
commit-hash: f9a839ea0c4c4885a5366d877a75ad3525bbab5e
commit-date: 2021-09-11
host: x86_64-unknown-linux-gnu
release: 1.56.0-beta.2
LLVM version: 13.0.0

Backtrace

<backtrace>

@rustbot label +A-edition-2021

[m-ou-se]

Thanks for your bug report.

This is expected with Cargo's new resolver = "2". See the note that cargo fix produced:

note: Switching to Edition 2021 will enable the use of the version 2 feature resolver in Cargo. This may cause some dependencies to be built with fewer features enabled than previously. More information about the resolver changes may be found at https://doc.rust-lang.org/nightly/edition-guide/rust-2021/default-cargo-resolver.html
When building the following dependencies, the given features will no longer be used:

diesel v1.4.7 (as host dependency) removed features: 32-column-tables, bitflags, default, postgres, pq-sys, with-deprecated

note: This project appears to use both diesel and diesel_migrations. These packages have a known issue where the build may fail due to the version 2 resolver preventing feature unification between those two packages. See rust-lang/cargo#9450 for some potential workarounds.

[...]

note: This error may be due to an interaction between diesel and Cargo's new feature resolver. Some workarounds you may want to consider:

• Add a build-dependency in Cargo.toml on diesel to force Cargo to add the appropriate features. This may look something like this:

   [build-dependencies]
   diesel = { version = "1.4.7", features = ["postgres"] }

• Try using the previous resolver by setting resolver = "1" in Cargo.toml (see https://doc.rust-lang.org/cargo/reference/resolver.html#resolver-versions for more information).

But I'm guessing you're aware of that, considering you reported that issue that's linked there, rust-lang/cargo#9450, and you're the one who suggested adding this very notice and the [build-dependencies] suggestion.

[weiznich]

Yes I'm well aware of this. I've raised this bug report because I do not feel that these messages are on the level that is expected for the edition migration mechanism. As far as I'm aware most of them where written with resolver = "2" in mind not with the automatic edition transition. I feel at least they should be improved. After all the corresponding RFC states that updating to a new edition should be possible by just doing the commands listed above.

(That all written it is unfortunately not possible to fix that on diesels side as fixing this does require a major version bump. At least in my opinion this is very unfortunate that such a change is "forced" by the language itself without offering the necessary tools to circumvent those issues at library level.)

[m-ou-se]

In an ideal world cargo fix --edition would just fix the problem and either add the corresponding section to Cargo.toml

I'm assuming that was not done that automatically because it's hard to make sure it's correct. @ehuss, I'm assuming you know more about this?

After all the corresponding RFC states that updating to a new edition should be possible by just doing the commands listed above.

Our goal with the automatic migrations was never to be fully perfect. The edition blog post states: "The automated migrations are not necessarily perfect: there might be some corner cases where manual changes are still required."
We aimed for at least 95%+ of crates to be able to automatically migrate without any manual intervention, and have reached a much higher number. The diesel-related failures found during the crater runs amounted to 0.07% of the tested crates. All resolver-related problems totaled to less than 0.5%, of which almost all are an old version of mime_guess not passing the unicase feature on to a dependency, which has already been solved in a patch release more than a year ago.

Cargo already warns about all resolver related changes during the migration, and for diesel errors even explains the issue and suggests the Cargo.toml change that's necessary, which you've suggested multiple times as an acceptable solution.

We can try to improve cargo fix --edition even further and make even more edge cases migrate automatically, and I'd definitely welcome such contributions. But I think we're already far past the point of deminishing returns. Especially considering this is a volunteer project with very limited resources.

As another show stop measurement cargo fix --edition should emit the same warning as cargo check already does. As the messaging currently reads it's unclear if the build will fail afterwards or not. Only cargo check gives the user a clear indication how to actually fix that problem

Feel free to suggest a better message to display from cargo fix --edition. The current message is here. Feel free to send a PR, or just leave your suggested improvement as a comment here.

As a medium term thing: Rust needs to provide better tools to bundle proc macro crates and other crates.

Agreed. I'd be very happy to see a concrete proposal for that. :)

[ehuss]

In an ideal world cargo fix --edition would just fix the problem and either add the corresponding section to Cargo.toml

Yea, unfortunately we don't have the capability to modify TOML files in cargo at this time.

[weiznich]

Our goal with the automatic migrations was never to be fully perfect. The edition blog post states: "The automated migrations are not necessarily perfect: there might be some corner cases where manual changes are still required."
We aimed for at least 95%+ of crates to be able to automatically migrate without any manual intervention, and have reached a much higher number. The diesel-related failures found during the crater runs amounted to 0.07% of the tested crates. All resolver-related problems totaled to less than 0.5%, of which almost all are an old version of mime_guess not passing the unicase feature on to a dependency, which has already been solved in a patch release more than a year ago.

Thanks for clarifying that. I've looked again into the corresponding RFC and you are right, both state that automated migrations should only work in most of the cases. Doing a crater run is there definitively a way to measure the breakage, but I'm not sure if that's the correct interpretation of the result numbers. The combination diesel + diesel_migrations tend to get used in binary rust project, so server side applications and such stuff. I assume that this code large parts of code dependent on both crates is not available on crates.io. For example crates.io itself depends on diesel and diesel_migrations and is not available as crate on crates.io. There is likely much more non public code with the same problem, so I assume that the real numbers are quite a bit larger here.

That written another thing I noticed while reading RFC 2052 and RFC 3085 is that RFC 2052 states the following:

Hard constraints

TL;DR: Warning-free code on edition N must compile on edition N+1 and have the same behavior.

(Source)

As of now that's not the case for the example provided. It compiles without any warning using the 2018 edition, but fails to compile with the 2021 edition.

Cargo already warns about all resolver related changes during the migration, and for diesel errors even explains the issue and suggests the Cargo.toml change that's necessary, which you've suggested multiple times as an acceptable solution.

Let me clarify that I only suggested that this is an acceptable workaround for the situation where people manually switch to the new resolver = "2" behaviour, not for automatically switching that as part of the much more prominent edition release.

Especially considering this is a volunteer project with very limited resources.

It's totally understandable to have such an approach but please keep in mind that you only shift the work from one volunteer project (cargo and the edition upgrade) to another one (diesel). Additionally you are doing this without given us any tool to handle this situation in any way, at least not without releasing a breaking change release at least while using this proposed solution.

Feel free to suggest a better message to display from cargo fix --edition. The current message is here. Feel free to send a PR, or just leave your suggested improvement as a comment here.

It would already be an improvement if cargo fix would emit the same message as cargo check later on. The later message provides a clear indication on what needs to be done, while the first message only says that there may be some problem. I feel that just require to set the resolver = "1/2" explicitly on the 2021 edition (and default to resolver = "2" for new projects) would likely be a better solution.

[m-ou-se]

I assume that this code large parts of code dependent on both crates is not available on crates.io. For example crates.io itself depends on diesel and diesel_migrations and is not available as crate on crates.io.

Note that next to all the crates from crates.io, crater also tests all crates from public GitHub repositories that have a Cargo.lock file. That second group is actually the majority. (Crater tests about 170k-180k crates. crates.io has 67k crates.)

[weiznich]

I've found a way to fix this in diesel without issuing a breaking release. It would be great if the message in cargo fix could be adjusted so that it does not show up if the diesel version is >= 1.4.8. Otherwise the message should prompt the user to update diesel to the latest release.

[weiznich]

I've issued a new diesel (1.4.8) to address this. Would be great to get the messages updated on cargo/rusts side till the next release. If someone points me into the right direction (reading where to change that + where to open a PR) I might find some time to work on this this week.

[ehuss]

That's good news! Are you suggesting that the messages can be removed? Here are the two places:

• For fix: report_maybe_diesel
• For normal builds: back_compat_notice

[weiznich]

I think it's better to change the message in such a way that it just suggests updating diesel instead of completely removing the message. I've opened a PR(rust-lang/cargo#9927) for this.