Add UploadWithResponseAsync API #4062

GarrettBeatty · 2025-10-18T15:42:34Z

Stacked PRs:

Description

This change creates a new api UploadWithResponseAsync. It is the same as the existing UploadAsync api but it returns the TransferUtilityUploadResponse object.

How

In order to re-use as much as the existing code as possible, I made BaseCommand take in a generic which is the return type of the ExecuteAsync function. All commands UploadCommand, DownloadCommand, etc will pass in their expected return type TransferUtilityUploadResponse, TransferUTilityDownloadResponse, etc.

In order to make this change in one go, I had to create place holder classes for TransferUtilityDownloadDirectoryResponse and other responses. These will eventually be populated in future PRs

Motivation and Context

To adhere to the SEP

Testing

1.dry run - d704b36b-7329-42dc-80e0-525a2f82bbcc pass
2. Integration tests

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist

My code follows the code style of this project
My change requires a change to the documentation
I have updated the documentation accordingly
I have read the README document
I have added tests to cover my changes
All new and existing tests passed

License

I confirm that this pull request can be released under the Apache 2 license

stack-info: PR: #4062, branch: GarrettBeatty/stacked/4

Copilot

Pull Request Overview

Adds typed response support to S3 TransferUtility and introduces new UploadWithResponseAsync APIs that return upload metadata (e.g., ETag). Refactors internal command pipeline to a generic BaseCommand pattern and adds response types for upload/download/stream/directory/abort operations.

Add UploadWithResponseAsync overloads on TransferUtility returning TransferUtilityUploadResponse
Introduce BaseCommand and update internal commands to return typed responses
Add response DTOs for directory/download/open stream/abort operations and update dev config to minor

Reviewed Changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
sdk/src/Services/S3/Custom/Transfer/_async/TransferUtility.async.cs	Adds UploadWithResponseAsync overloads and wires typed command execution/return.
sdk/src/Services/S3/Custom/Transfer/TransferUtilityUploadDirectoryResponse.cs	Adds response DTO for upload directory (placeholder).
sdk/src/Services/S3/Custom/Transfer/TransferUtilityOpenStreamResponse.cs	Adds response DTO for open stream (currently empty).
sdk/src/Services/S3/Custom/Transfer/TransferUtilityDownloadResponse.cs	Adds response DTO for download (placeholder).
sdk/src/Services/S3/Custom/Transfer/TransferUtilityDownloadDirectoryResponse.cs	Adds response DTO for download directory (namespace differs from others).
sdk/src/Services/S3/Custom/Transfer/TransferUtilityAbortMultipartUploadsResponse.cs	Adds response DTO for abort multipart uploads (placeholder).
sdk/src/Services/S3/Custom/Transfer/TransferUtility.cs	Updates internal GetUploadCommand to typed BaseCommand.
sdk/src/Services/S3/Custom/Transfer/Internal/_bcl+netstandard/UploadDirectoryCommand.cs	Returns TransferUtilityUploadDirectoryResponse from ExecuteAsync.
sdk/src/Services/S3/Custom/Transfer/Internal/_bcl+netstandard/DownloadDirectoryCommand.cs	Returns TransferUtilityDownloadDirectoryResponse from ExecuteAsync.
sdk/src/Services/S3/Custom/Transfer/Internal/_async/SimpleUploadCommand.async.cs	Returns TransferUtilityUploadResponse and fires completed event.
sdk/src/Services/S3/Custom/Transfer/Internal/_async/OpenStreamCommand.async.cs	Returns TransferUtilityOpenStreamResponse (currently not carrying stream).
sdk/src/Services/S3/Custom/Transfer/Internal/_async/MultipartUploadCommand.async.cs	Returns TransferUtilityUploadResponse; unseekable path returns mapped response.
sdk/src/Services/S3/Custom/Transfer/Internal/_async/DownloadCommand.async.cs	Returns TransferUtilityDownloadResponse after download completes.
sdk/src/Services/S3/Custom/Transfer/Internal/_async/BaseCommand.async.cs	Makes BaseCommand generic and updates ExecuteCommandAsync signature.
sdk/src/Services/S3/Custom/Transfer/Internal/_async/AbortMultipartUploadsCommand.async.cs	Returns TransferUtilityAbortMultipartUploadsResponse from ExecuteAsync.
sdk/src/Services/S3/Custom/Transfer/Internal/UploadDirectoryCommand.cs	Type aligns with BaseCommand.
sdk/src/Services/S3/Custom/Transfer/Internal/SimpleUploadCommand.cs	Type aligns with BaseCommand.
sdk/src/Services/S3/Custom/Transfer/Internal/OpenStreamCommand.cs	Type aligns with BaseCommand; removes Return override.
sdk/src/Services/S3/Custom/Transfer/Internal/MultipartUploadCommand.cs	Type aligns with BaseCommand.
sdk/src/Services/S3/Custom/Transfer/Internal/DownloadDirectoryCommand.cs	Type aligns with BaseCommand.
sdk/src/Services/S3/Custom/Transfer/Internal/DownloadCommand.cs	Type aligns with BaseCommand.
sdk/src/Services/S3/Custom/Transfer/Internal/BaseCommand.cs	Converts BaseCommand to generic and removes Return property.
sdk/src/Services/S3/Custom/Transfer/Internal/AbortMultipartUploadsCommand.cs	Type aligns with BaseCommand.
generator/.DevConfigs/77d980ad-8f58-4f2e-97f8-d2c8c5ba3732.json	Adds dev config (minor) noting new UploadWithResponseAsync APIs and response-mapping enhancements.

sdk/src/Services/S3/Custom/Transfer/TransferUtilityDownloadDirectoryResponse.cs

sdk/src/Services/S3/Custom/Transfer/TransferUtilityOpenStreamResponse.cs

sdk/src/Services/S3/Custom/Transfer/Internal/_async/OpenStreamCommand.async.cs

sdk/src/Services/S3/Custom/Transfer/TransferUtilityOpenStreamResponse.cs

sdk/src/Services/S3/Custom/Transfer/_async/TransferUtility.async.cs

stack-info: PR: #4062, branch: GarrettBeatty/stacked/4

GarrettBeatty · 2025-10-20T14:11:08Z

@philasmar this pr is still in draft but wanted to get your feedback on this approach. seems to be the easiest/best way to add this new api and re-use the existing functionality but using generics. i havent filled in the details of each response object yet (they are placeholder classes). was going to do that after

GarrettBeatty · 2025-10-20T14:11:39Z

sdk/src/Services/S3/Custom/Transfer/Internal/BaseCommand.cs

+    /// <typeparam name="TResponse">Type of response returned by the command</typeparam>
+    internal abstract partial class BaseCommand<TResponse> where TResponse : class
    {
-        public virtual object Return


this is not being used anywhere so removing it to avoid confusion

GarrettBeatty · 2025-10-24T14:43:58Z

sdk/src/Services/S3/Custom/_async/AmazonS3Client.Extensions.cs

-            };
-            InternalSDKUtils.ApplyValuesV2(request, additionalProperties);
-            return transfer.UploadAsync(request, cancellationToken);
+            return UploadObjectFromStreamInternalAsync(bucketName, objectKey, stream, additionalProperties, cancellationToken);


made a helper function here so that we could use the new uploadwithresponseasync api but then still have the same function signature of returning Task only (not Task<TransferUtilityUploadResponse>

GarrettBeatty · 2025-10-24T14:45:18Z

sdk/src/Services/S3/Custom/Transfer/Internal/_async/AbortMultipartUploadsCommand.async.cs

                await WhenAllOrFirstExceptionAsync(pendingTasks,cancellationToken)
                    .ConfigureAwait(continueOnCapturedContext: false);
+
+                return new TransferUtilityAbortMultipartUploadsResponse();


i dont think we plan on populating this at all FYI

GarrettBeatty · 2025-10-24T16:53:44Z

sdk/src/Services/S3/Custom/Transfer/_bcl+netstandard/TransferUtility.sync.cs

            try
            {
-                UploadAsync(filePath, bucketName).Wait();
+                UploadWithResponseAsync(filePath, bucketName).Wait();


actually i just realized this should say as UploadAsync because otherwise this would eventually create a span with UploadAsyncWithResponse instead of UploadAsync. will update that in next revision. same in other places below

stack-info: PR: #4075, branch: GarrettBeatty/stacked/8

stack-info: PR: #4076, branch: GarrettBeatty/stacked/9

stack-info: PR: #4062, branch: GarrettBeatty/stacked/4 add obsolete functions

philasmar · 2025-10-28T15:21:40Z

sdk/src/Services/S3/Custom/Transfer/_async/TransferUtility.async.cs

        ///     A cancellation token that can be used by other objects or threads to receive notice of cancellation.
        /// </param>
        /// <returns>The task object representing the asynchronous operation.</returns>
+        [Obsolete("Use UploadWithResponseAsync instead which allows you to upload files and also returns response metadata.")]


why do we need to mark these as obsolete?

philasmar · 2025-10-28T15:22:12Z

sdk/src/Services/S3/Custom/Transfer/_async/TransferUtility.async.cs

        ///     A cancellation token that can be used by other objects or threads to receive notice of cancellation.
        /// </param>
        /// <returns>The task object representing the asynchronous operation.</returns>
+        [Obsolete("Use UploadWithResponseAsync instead which allows you to upload files and also returns response metadata.")]


same question. the obsolete reason is not really convincing enough to mark the method as obsolete.

we want to encourage users to use the new functions. we are also creating downloadwithresponse (Which will have multi part download) so having them use the "withResponse" functions of both upload/download makes sense to me

philasmar · 2025-10-28T15:22:20Z

sdk/src/Services/S3/Custom/Transfer/_async/TransferUtility.async.cs

        ///     A cancellation token that can be used by other objects or threads to receive notice of cancellation.
        /// </param>
        /// <returns>The task object representing the asynchronous operation.</returns>
+        [Obsolete("Use UploadWithResponseAsync instead which allows you to upload files and also returns response metadata.")]


same comment

philasmar · 2025-10-28T15:22:25Z

sdk/src/Services/S3/Custom/Transfer/_async/TransferUtility.async.cs

        ///     A cancellation token that can be used by other objects or threads to receive notice of cancellation.
        /// </param>
        /// <returns>The task object representing the asynchronous operation.</returns>
+        [Obsolete("Use UploadWithResponseAsync instead which allows you to upload files and also returns response metadata.")]


same comment

peterrsongg

we should remove the return null statements, everything else are just small nits

peterrsongg · 2025-10-27T21:13:29Z

sdk/src/Services/S3/Custom/Transfer/_async/TransferUtility.async.cs

        }
+
+        /// <summary>
+        /// 	Uploads the specified file and returns response metadata including ETag, encryption details, and version information.


nit: spacing in the comments

peterrsongg · 2025-10-28T18:57:27Z

sdk/src/Services/S3/Custom/_async/AmazonS3Client.Extensions.cs

+
+        #region Internal Helper Methods
+
+        private async Task UploadObjectFromStreamInternalAsync(string bucketName, string objectKey, Stream stream, IDictionary<string, object> additionalProperties, CancellationToken cancellationToken)


nit: can we pass in TURequest object instead so the number of params goes down? best practice to not have too many params.

sdk/src/Services/S3/Custom/Transfer/_bcl+netstandard/TransferUtility.sync.cs

GarrettBeatty · 2025-10-29T15:11:10Z

opened this pr #4085

GarrettBeatty added a commit that referenced this pull request Oct 18, 2025

uploadwithresponse api

21a5f4f

stack-info: PR: #4062, branch: GarrettBeatty/stacked/4

GarrettBeatty force-pushed the GarrettBeatty/stacked/4 branch from 30b6e85 to 21a5f4f Compare October 18, 2025 15:42

This was referenced Oct 18, 2025

Add Progress listeners for initiated, complete, and failed for simple upload #4059

Merged

Add mapping of CompletemultipartUploadResponse to TransferUtilityUploadResponse #4060

Merged

Add multipartupload lifecycle tracking #4061

Merged

GarrettBeatty requested a review from Copilot October 18, 2025 19:01

Copilot AI reviewed Oct 18, 2025

View reviewed changes

GarrettBeatty changed the base branch from GarrettBeatty/stacked/3 to gcbeatty/tuuploadresponse October 18, 2025 19:54

GarrettBeatty added a commit that referenced this pull request Oct 18, 2025

uploadwithresponse api

e27d144

stack-info: PR: #4062, branch: GarrettBeatty/stacked/4

GarrettBeatty force-pushed the GarrettBeatty/stacked/4 branch from 21a5f4f to e27d144 Compare October 18, 2025 19:54

GarrettBeatty changed the base branch from gcbeatty/tuuploadresponse to GarrettBeatty/stacked/3 October 18, 2025 19:54

GarrettBeatty changed the base branch from GarrettBeatty/stacked/3 to gcbeatty/tuuploadresponse October 19, 2025 18:13

GarrettBeatty force-pushed the GarrettBeatty/stacked/4 branch from e27d144 to c89de03 Compare October 19, 2025 18:13

GarrettBeatty added a commit that referenced this pull request Oct 19, 2025

uploadwithresponse api

c89de03

stack-info: PR: #4062, branch: GarrettBeatty/stacked/4

GarrettBeatty changed the base branch from gcbeatty/tuuploadresponse to GarrettBeatty/stacked/3 October 19, 2025 18:13

GarrettBeatty changed the base branch from GarrettBeatty/stacked/3 to gcbeatty/tuuploadresponse October 19, 2025 18:15

GarrettBeatty force-pushed the GarrettBeatty/stacked/4 branch from c89de03 to 8d603df Compare October 19, 2025 18:15

GarrettBeatty added a commit that referenced this pull request Oct 19, 2025

uploadwithresponse api

8d603df

stack-info: PR: #4062, branch: GarrettBeatty/stacked/4

GarrettBeatty changed the base branch from gcbeatty/tuuploadresponse to GarrettBeatty/stacked/3 October 19, 2025 18:15

GarrettBeatty changed the base branch from GarrettBeatty/stacked/3 to gcbeatty/tuuploadresponse October 20, 2025 13:38

GarrettBeatty added a commit that referenced this pull request Oct 20, 2025

uploadwithresponse api

622b802

stack-info: PR: #4062, branch: GarrettBeatty/stacked/4

GarrettBeatty force-pushed the GarrettBeatty/stacked/4 branch from 8d603df to 622b802 Compare October 20, 2025 13:38

GarrettBeatty changed the base branch from gcbeatty/tuuploadresponse to GarrettBeatty/stacked/3 October 20, 2025 13:38

GarrettBeatty requested a review from philasmar October 20, 2025 13:43

GarrettBeatty changed the base branch from GarrettBeatty/stacked/3 to feature/transfermanager October 20, 2025 14:06

GarrettBeatty added a commit that referenced this pull request Oct 20, 2025

uploadwithresponse api

bb1413b

stack-info: PR: #4062, branch: GarrettBeatty/stacked/4

GarrettBeatty force-pushed the GarrettBeatty/stacked/4 branch from 622b802 to bb1413b Compare October 20, 2025 14:06

GarrettBeatty changed the base branch from feature/transfermanager to GarrettBeatty/stacked/3 October 20, 2025 14:06

GarrettBeatty commented Oct 20, 2025

View reviewed changes

GarrettBeatty changed the base branch from GarrettBeatty/stacked/9 to feature/transfermanager October 24, 2025 13:48

GarrettBeatty force-pushed the GarrettBeatty/stacked/4 branch from a3dfaec to d3860eb Compare October 24, 2025 13:48

GarrettBeatty changed the base branch from feature/transfermanager to GarrettBeatty/stacked/9 October 24, 2025 13:49

GarrettBeatty commented Oct 24, 2025

View reviewed changes

GarrettBeatty changed the base branch from GarrettBeatty/stacked/9 to feature/transfermanager October 24, 2025 14:57

GarrettBeatty force-pushed the GarrettBeatty/stacked/4 branch from d3860eb to 6d18ba4 Compare October 24, 2025 14:57

GarrettBeatty changed the base branch from feature/transfermanager to GarrettBeatty/stacked/9 October 24, 2025 14:57

GarrettBeatty changed the base branch from GarrettBeatty/stacked/9 to feature/transfermanager October 24, 2025 15:17

GarrettBeatty changed the base branch from feature/transfermanager to GarrettBeatty/stacked/9 October 24, 2025 15:18

GarrettBeatty changed the base branch from GarrettBeatty/stacked/9 to feature/transfermanager October 24, 2025 15:46

GarrettBeatty changed the base branch from feature/transfermanager to GarrettBeatty/stacked/9 October 24, 2025 15:46

GarrettBeatty changed the base branch from GarrettBeatty/stacked/9 to feature/transfermanager October 24, 2025 16:19

GarrettBeatty changed the base branch from feature/transfermanager to GarrettBeatty/stacked/9 October 24, 2025 16:19

GarrettBeatty marked this pull request as ready for review October 24, 2025 16:38

GarrettBeatty commented Oct 24, 2025

View reviewed changes

GarrettBeatty changed the base branch from GarrettBeatty/stacked/9 to feature/transfermanager October 24, 2025 22:24

GarrettBeatty force-pushed the GarrettBeatty/stacked/4 branch from 6d18ba4 to 73b483f Compare October 24, 2025 22:24

GarrettBeatty changed the base branch from feature/transfermanager to GarrettBeatty/stacked/9 October 24, 2025 22:24

GarrettBeatty added 3 commits October 28, 2025 09:11

Add DownloadResponse mapping

11bd069

stack-info: PR: #4075, branch: GarrettBeatty/stacked/8

Add GetObjectResponse to TransferUtilityOpenStreamResponse mapping.

b5e77c5

stack-info: PR: #4076, branch: GarrettBeatty/stacked/9

Add UploadWithResponseAsync API

60fdeae

stack-info: PR: #4062, branch: GarrettBeatty/stacked/4 add obsolete functions

GarrettBeatty changed the base branch from GarrettBeatty/stacked/9 to feature/transfermanager October 28, 2025 13:18

GarrettBeatty force-pushed the GarrettBeatty/stacked/4 branch from 73b483f to 60fdeae Compare October 28, 2025 13:18

GarrettBeatty changed the base branch from feature/transfermanager to GarrettBeatty/stacked/9 October 28, 2025 13:18

philasmar reviewed Oct 28, 2025

View reviewed changes

peterrsongg requested changes Oct 28, 2025

View reviewed changes

GarrettBeatty force-pushed the GarrettBeatty/stacked/9 branch from b5e77c5 to ff5fbf0 Compare October 29, 2025 15:10

GarrettBeatty closed this Oct 29, 2025


		#region Internal Helper Methods

		private async Task UploadObjectFromStreamInternalAsync(string bucketName, string objectKey, Stream stream, IDictionary<string, object> additionalProperties, CancellationToken cancellationToken)

Add UploadWithResponseAsync API #4062

Add UploadWithResponseAsync API #4062

Uh oh!

Conversation

GarrettBeatty commented Oct 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

How

Motivation and Context

Testing

Types of changes

Checklist

License

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GarrettBeatty commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GarrettBeatty Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GarrettBeatty Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peterrsongg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

GarrettBeatty commented Oct 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

GarrettBeatty commented Oct 18, 2025 •

edited

Loading

GarrettBeatty commented Oct 20, 2025 •

edited

Loading

GarrettBeatty Oct 24, 2025 •

edited

Loading

GarrettBeatty Oct 24, 2025 •

edited

Loading