SDK build sporadically failing due to multiple pipelines trying to concurrently upload 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip'

[danmoseley]

This may possibly just corrupt the blob instead. @joeloff is investigating. We probably should prioritize fixing...

2021-08-11T22:57:06.4524916Z Attempt '1' for uploading chunk '20' of file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip'.
2021-08-11T22:57:06.4525531Z Generate new HttpRequest for uploading file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip', chunk '20' of '21'.
2021-08-11T22:57:06.4526168Z Start uploading file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip' to server, chunk '20'.
2021-08-11T22:57:06.4526978Z Finished upload chunk '20' of file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip', elapsed 511 (ms), response code 'Accepted'.
2021-08-11T22:57:06.4527796Z Attempt '1' for uploading chunk '21' of file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip'.
2021-08-11T22:57:06.4528585Z Generate new HttpRequest for uploading file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip', chunk '21' of '21'.
2021-08-11T22:57:06.4529202Z Start uploading file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip' to server, chunk '21'.
2021-08-11T22:57:06.4532145Z Chunk '21' attempt '1' of file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip' fail to send request to server. Error: Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException: Blob is incomplete (missing block). Blob: af03054df7faeb11b563501ac5b0ff56, Expected Offset: 0, Actual Offset: 83886080
 ---> System.InvalidOperationException: Blob is incomplete (missing block). Blob: af03054df7faeb11b563501ac5b0ff56, Expected Offset: 0, Actual Offset: 83886080
   --- End of inner exception stack trace ---
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.HandleResponseAsync(HttpResponseMessage response, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.FileContainer.Client.FileContainerHttpClient.UploadFileAsync(Int64 containerId, String itemPath, Stream fileStream, Byte[] contentId, Int64 fileLength, Boolean isGzipped, Guid scopeIdentifier, CancellationToken cancellationToken, Int32 chunkSize, Int32 chunkRetryTimes, Boolean uploadFirstChunk, Object userState).
2021-08-11T22:57:06.4535515Z Backoff 6.48 seconds before attempt '2' chunk '21' of file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip'.
2021-08-11T22:57:06.4536162Z Attempt '2' for uploading chunk '21' of file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip'.
2021-08-11T22:57:06.4536822Z Generate new HttpRequest for uploading file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip', chunk '21' of '21'.
2021-08-11T22:57:06.4537527Z Start uploading file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip' to server, chunk '21'.
2021-08-11T22:57:06.4539684Z Chunk '21' attempt '2' of file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip' fail to send request to server. Error: Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException: Bad Request
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.HandleResponseAsync(HttpResponseMessage response, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.FileContainer.Client.FileContainerHttpClient.UploadFileAsync(Int64 containerId, String itemPath, Stream fileStream, Byte[] contentId, Int64 fileLength, Boolean isGzipped, Guid scopeIdentifier, CancellationToken cancellationToken, Int32 chunkSize, Int32 chunkRetryTimes, Boolean uploadFirstChunk, Object userState).
2021-08-11T22:57:06.4542219Z Backoff 5.466 seconds before attempt '3' chunk '21' of file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip'.
2021-08-11T22:57:06.4542842Z Attempt '3' for uploading chunk '21' of file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip'.
2021-08-11T22:57:06.4543477Z Generate new HttpRequest for uploading file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip', chunk '21' of '21'.
2021-08-11T22:57:06.4544114Z Start uploading file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip' to server, chunk '21'.
2021-08-11T22:57:06.4544643Z ##[section]Async Command End: Upload Artifact

Same thing happened here:

#19582 (comment)

See also dotnet/core-eng#14015

@dotnet/domestic-cat

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[danmoseley]

This is happening regularly, eg., in the maestro ingestion today for runtime

https://dev.azure.com/dnceng/public/_build/results?buildId=1289894&view=logs&jobId=60adcdc4-a9f0-5668-20f7-b72181c68fb2&j=e98fba6a-08a4-5b7c-1ef9-6d6c234aaf57&t=1735c990-7aa9-5be6-fc9c-fceb366f4f34&s=ff05ad62-bb9a-53b6-ce9f-72f329a63e7c

https://dev.azure.com/dnceng/public/_build/results?buildId=1289894&view=logs&jobId=60adcdc4-a9f0-5668-20f7-b72181c68fb2&j=adc369b2-ee17-52c3-72b2-7129c9e8cda1&t=78acff75-e3a6-582d-ecc2-9f0bf523e0bc&s=ff05ad62-bb9a-53b6-ce9f-72f329a63e7c

both are the full framework ones -- probably because they take about the same time. Both are trying to upload the same blob 'BlobArtifacts/dotnet-toolset-langpack-6.0.100-ci.zip'

@marcpopMSFT this is probably important to look into because we don't know what is getting uploaded, whether it's corrupted etc. It's also causing retries which consume resources.

[joeloff]

I'll take a look today. We probably only need one or need to add the configuration property as a discriminator.

[joeloff]

So this is a zip that contains the full SDK. Looks like it's being used as a transport for dotnet/installer that we then unzip and harvest into the .NET SDK installer. Not sure why we have the langpack, because the normal zip contains the resource assemblies already.

@dsplaisted any concerns if we only publish this for the release build?

[joeloff]

I searched dotnet for dotnet-toolset-internal references. It only shows the SDK, but it's used in the installer repo, so now I'm not sure if I should trust the search. I'm trying to determine whether anyone else besides dotnet/installer depends on this artifact

[joeloff]

Chatted with a few folks. We can just turn off publishing on the debug leg for CI.

[lewing]

Saw this today https://dev.azure.com/dnceng/public/_build/results?buildId=1298276&view=logs&j=ffcccca8-8d4a-5f1e-019c-64de63d1f0d8&t=b45037e0-dc32-5dd3-9457-1b637e753ec3&l=881