make build.zig.zon contain all urls, including mirrors, and hashes of entire dependency tree

[andrewrk]

Extracted from #14265.

The motivation for this is faster dependency fetching, avoiding round-trips from the network.

The idea here is that, after fetching all dependencies, recursively, zig would edit the user's build.zig.zon file and automatically add a section that contains all urls & hashes (including mirrors) for the entire dependency tree. It might look something like this:

.{
    // ...
    .mirrors = .{
        .@"sha256=c9b30cffc40999d2c078ff350cbcee642970a224fe123c756d0892f876cf1aae" = .{
            "https://example.com/1.tar.gz",
            "https://example.com/2.tar.gz",
        },
    },
}

In this case it matches exactly with #14292 which could be nice, or, on the other hand, it might be important to make a distinction between automatically added ones (which should also be removed automatically) or manually added ones (which should always be preserved).

Depending on how #14288 is resolved with regards to conflict resolution, this section would perhaps be combined with more tooling-managed data about each dependency, such as dependency overrides:

.{
    // ...
    .config = .{
        .@"sha256=c9b30cffc40999d2c078ff350cbcee642970a224fe123c756d0892f876cf1aae" = .{
            .url = .{
                "https://example.com/1.tar.gz",
                "https://example.com/2.tar.gz",
            },
            .overrides = .{
                .libz = "sha256=9ba4f49895b174a3f918d489238acbc146bd393575062b2e3be33488b688e36f",
            },
        },
    },
}

When combined with #14294, this may additionally need to float the fetch plugins to the top along with mirrors.

[nektro]

rather than mixing manual and automatic ones and since its going to contain info on the entire tree, imo it should be its own .zon file

[deflock]

I doubt it's a good idea to modify user's build.zig.zon, in some crazy setups it may be mounted/bound as readonly for some reason, when using containers or something like this

[andrewrk]

It's already planned to modify build.zig.zon as part of #14288, for example when a package dependency is added or removed.

[deflock]

Looks like we need a QA stream with describing complete package management workflow by writing an RFC there😅

[loic-sharma]

Copying my comment: #14290 (comment)

I used to work on NuGet, .NET's package manager. NuGet originally put the entire dependency tree in its packages.config file. This was a huge mistake. We later migrated to a new format, PackageReference, where you only list your project's direct dependencies. This migration made a significant difference in developer satisfaction (measured through HaTS).

more complicated logic for fetching

Very true, but, you're offloading this complexity to the developer. .NET developers complained that packages.config was unmanageable. It was difficult to tell if a dependency could be safely removed. Also, you'd need to understand your dependency tree if you wanted to update a single dependency. Excellent IDE tooling was not enough to offset this pain.

not only from a performance perspective

Performance was a top concern for NuGet's migration: NuGet runs in every .NET build, including Visual Studio's design time builds that power Intellisense. We were able to make this fast by:

1. Use minimum version selection. This makes your dependency graph stable for a given set of direct dependencies. In other words, you only need to download packages if the developer adds/updates their direct dependencies.
2. Optimize for the "no-op" case. NuGet is blazing fast if your direct dependencies haven't changed.

EDIT: NuGet does store its full dependency tree in its lock file. However, that's a separate file that's not meant to be edited by the developer.

[mlugg]

While caching this information for performance does make sense, I do think it would be better for the full dependency tree to be cached in a different file, maybe build-cache.zon or similar for a few reasons. This is significantly more manageable for the user (particularly if they want to edit the file directly rather than using tooling, which I definitely think should be a supported use case), gives cleaner repository commit history (you can just ignore changes to build-cache.zon, you only need to actually worry about build.zon! also yeah i want to bikeshed that name i'm not a fan of build.zig.zon), and makes it easier to identify old dependencies and other cruft in your build.zon.