Support retry mechanism for Git operations when server returns 429 Too Many Requests

[BiagioFesta]

Problem

Cargo's net.retry configuration currently doesn't affect Git operations when they fail.

In enterprise environments, it's common to have internal Git servers hosting private dependencies. These servers often implement rate limiting to prevent overload, especially when many builds happen in parallel (CI/CD pipelines, multiple developers, etc.).

Currently, when the Git server returns a 429, the build fails immediately:

$ cargo build
remote: Too Many Requests
fatal: unable to access 'https://myserver:8888/git/': The requested URL returned error: 429
error: failed to get my_beauty_lib as a dependency of package ...

Am I wondering whether it could be an option to allow cargo git invocation retry in this case.

Proposed Solution

Extend Cargo's net.retry configuration (or introduce a new net.git-retry option) to handle Git operation retries, particularly for rate-limit (429) responses.

Notes

IINM, even with net.git-fetch-with-cli, it seems this is not possible, as it seems git does not support retry config; so the caller (cargo in this case) is required to implement the retry mechanism.

[weihanglo]

With git2 or gitoxide backend, you shall see something like

[WARNING] spurious network error (3 tries remaining): failed to get successful HTTP response from `/some/api/` ([..]), got 500

But only when the error matches HttpNotSuccessful.

cargo/src/cargo/util/network/retry.rs

Line 107 in 6d221ef

let err = e.downcast_ref::<HttpNotSuccessful>();

This may need more exploration on libgit2/gitoxide to cover more HTTP-like network errors.

There is no retry mechanism for net.git-fetch-with-cli at this moment btw.

@rustbot label +S-needs-design -S-triage

[BiagioFesta]

I'm wondering if I could propose a simple patch to add retry support for fetch_with_cli.

From what I can tell, the retry logic should be placed here:

cargo/src/cargo/sources/git/utils.rs

Line 1113 in b624a44

cmd.exec()?;

(Though I'm still unsure whether to wrap the entire cmd construction or just the exec call)

That said, this alone isn't sufficient, because with_retry (used for git2 and gitoxide) only retries if the returned error satisfies a specific condition:

cargo/src/cargo/util/network/retry.rs

Line 187 in b624a44

fn maybe_spurious(err: &Error) -> bool {

I see three possible simple approaches (by "simple" I mean solutions that avoid larger design changes):

1. (Simplest, but naive): Retry on all ProcessErrors:

fn maybe_spurious(...) {
   // ...
  if err.downcast_ref::<cargo_util::ProcessError>().is_some() {
      return true;
  }

2. Add an is_spurious property to ProcessError and retry only if true:

This would be more precise, but would require modifying all the places where ProcessError is constructed. Moreover, since ProcessError lives in the cargo_util crate, this might introduce undesired coupling or broaden the scope too much.

3. Add an always_spurious argument flag to the function with_retry:

This would act as an override, allowing the retry to occur either when the error is considered spurious or when the caller deems the operation always retryable.

[weihanglo]

Is there a reliable way to tell that an error from git CLI is networking issue? We might be able to wrap ProcessError with HttpNotSuccessful error.

[BiagioFesta]

No, or at least I haven't found any (reliable).

That was my first search attempt, but it appears that the git fetch command doesn't have standardized exit codes. It always exits with 128 regardless of the error reason

[weihanglo]

Maybe we can do some basic string matching in the output? If found nothing just don't retry.

[BiagioFesta]

I'm curious about the string matching approach, while it could work, I wonder about its reliability across different git versions, time, and localizations (language).

Would it be simpler to retry all git fetch failures? The main tradeoff seems to be retrying non-network issues, but fetch command is generally for networking interactions, and non-network failures typically fail fast.
Do you see any particular issue in that?

[weihanglo]

Nah I forgot Git supports locales. Retrying for all seems simpler yeah. Just we might want to deal with this as well: #14964