DispatchFailure: HTTP/2 REFUSED_STREAM should be retried #858
Description
Describe the bug
We regularly get an HTTP/2 REFUSED_STREAM error in production when deleting S3 blobs. This error means that a request should be retried, so I'm opening an issue based on the following warning that was emitted:
aws_smithy_client::hyper_ext
unrecognized error from Hyper. If this error should be retried, please file an issue.
{err: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic (hyper::Error(Http2, Error { kind: Reset(StreamId(243), REFUSED_STREAM, Remote) }))}
See https://datatracker.ietf.org/doc/html/rfc9113#REFUSED_STREAM and https://datatracker.ietf.org/doc/html/rfc9113#Reliability saying that such requests can be safely retried.
Expected Behavior
When calling DispatchFailure::other, it should have returned Some. Unclear which ErrorKind exactly, but probably ErrorKind::ServerError.
Current Behavior
When calling DispatchFailure::other, it returns None.
Reproduction Steps
I haven't found a way to reproduce this yet, I can only see it in production sometimes when using aws-sdk-s3 against the google cloud storage API.
Possible Solution
Treat the REFUSED_STREAM error as aws_smithy_types::retry::ErrorKind::ServerError
Additional Information/Context
No response
Version
$ cargo tree | grep aws -
├── aws-sdk-s3 v0.28.0
│ ├── aws-credential-types v0.55.3
│ │ ├── aws-smithy-async v0.55.3
│ │ ├── aws-smithy-types v0.55.3
│ ├── aws-endpoint v0.55.3
│ │ ├── aws-smithy-http v0.55.3
│ │ │ ├── aws-smithy-eventstream v0.55.3
│ │ │ │ ├── aws-smithy-types v0.55.3 (*)
│ │ │ ├── aws-smithy-types v0.55.3 (*)
│ │ ├── aws-smithy-types v0.55.3 (*)
│ │ ├── aws-types v0.55.3
│ │ │ ├── aws-credential-types v0.55.3 (*)
│ │ │ ├── aws-smithy-async v0.55.3 (*)
│ │ │ ├── aws-smithy-client v0.55.3
│ │ │ │ ├── aws-smithy-async v0.55.3 (*)
│ │ │ │ ├── aws-smithy-http v0.55.3 (*)
│ │ │ │ ├── aws-smithy-http-tower v0.55.3
│ │ │ │ │ ├── aws-smithy-http v0.55.3 (*)
│ │ │ │ │ ├── aws-smithy-types v0.55.3 (*)
│ │ │ │ ├── aws-smithy-types v0.55.3 (*)
│ │ │ ├── aws-smithy-http v0.55.3 (*)
│ │ │ ├── aws-smithy-types v0.55.3 (*)
│ ├── aws-http v0.55.3
│ │ ├── aws-credential-types v0.55.3 (*)
│ │ ├── aws-smithy-http v0.55.3 (*)
│ │ ├── aws-smithy-types v0.55.3 (*)
│ │ ├── aws-types v0.55.3 (*)
│ ├── aws-sig-auth v0.55.3
│ │ ├── aws-credential-types v0.55.3 (*)
│ │ ├── aws-sigv4 v0.55.3
│ │ │ ├── aws-smithy-eventstream v0.55.3 (*)
│ │ │ ├── aws-smithy-http v0.55.3 (*)
│ │ ├── aws-smithy-eventstream v0.55.3 (*)
│ │ ├── aws-smithy-http v0.55.3 (*)
│ │ ├── aws-types v0.55.3 (*)
│ ├── aws-sigv4 v0.55.3 (*)
│ ├── aws-smithy-async v0.55.3 (*)
│ ├── aws-smithy-checksums v0.55.3
│ │ ├── aws-smithy-http v0.55.3 (*)
│ │ ├── aws-smithy-types v0.55.3 (*)
│ ├── aws-smithy-client v0.55.3 (*)
│ ├── aws-smithy-eventstream v0.55.3 (*)
│ ├── aws-smithy-http v0.55.3 (*)
│ ├── aws-smithy-http-tower v0.55.3 (*)
│ ├── aws-smithy-json v0.55.3
│ │ └── aws-smithy-types v0.55.3 (*)
│ ├── aws-smithy-types v0.55.3 (*)
│ ├── aws-smithy-xml v0.55.3
│ ├── aws-types v0.55.3 (*)
├── aws-smithy-async v0.55.3 (*)
├── aws-smithy-http v0.55.3 (*)
├── aws-smithy-types v0.55.3 (*)
├── aws-sdk-s3 v0.28.0 (*)
Environment details (OS name and version, etc.)
Docker on Linux (not really relevant)
Logs
As shown above:
aws_smithy_client::hyper_ext
unrecognized error from Hyper. If this error should be retried, please file an issue.
{err: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic (hyper::Error(Http2, Error { kind: Reset(StreamId(243), REFUSED_STREAM, Remote) }))}