Unable to clean up running docker containers

[jander-msft]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=300842
Build error leg or test failing: "Libraries Test Run release mono linux x64 Debug" and "Libraries Test Run release coreclr linux x64 Debug"
Pull request: #86813

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "Unable to clean up running docker containers",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=300842
Build error leg or test failing: "Libraries Test Run release mono linux x64 Debug" and "Libraries Test Run release coreclr linux x64 Debug"
Pull request: #86813

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "Unable to clean up running docker containers",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Author:	jander-msft
Assignees:	-
Labels:	area-Infrastructure-libraries
Milestone:	-

[jander-msft]

This is an Helix queue infra error that occurs when trying to run work items on (AlmaLinux.8.Amd64.Open)RedHat.7.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:almalinux-8-helix-amd64

All of the work items running in this queue have a console output similar to:

Console log: 'ComInterfaceGenerator.Tests' from job ab1b4172-455b-45b3-8942-574a86d2c8fc (redhat.7.amd64.open.rt) using docker image mcr.microsoft.com/dotnet-buildtools/prereqs:almalinux-8-helix-amd64 on a000wrn
running $HELIX_CORRELATION_PAYLOAD/scripts/ce392291899745e2ac937dda7673e2e5/execute.sh in /mnt/work/A40D0917/w/A06A08EB/e max 900 seconds

Output:
Unable to clean up running docker containers
Exit Code:-6

the run_client.py log has

2023-06-09T04:24:03.103Z    INFO    executor(113)   run Successfully downloaded work item payload
2023-06-09T04:24:03.103Z    INFO    interval(67)    set_abort_after_timer   Set the timer's abort_after to 960 seconds from now: abort_after = 2023-06-09 04:40:03.103779
2023-06-09T04:24:03.126Z    INFO    dockerhelper(67)    ensure_docker_running   Docker does not appear to be on this machine
2023-06-09T04:24:03.127Z    INFO    dockerhelper(76)    ensure_docker_running   Done.
2023-06-09T04:24:03.127Z    INFO    dockerhelper(22)    cleanup Cleaning up Docker on the machine
2023-06-09T04:24:03.129Z    ERROR   dockerhelper(33)    cleanup Error trying to clean running docker containers: "('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))"
2023-06-09T04:24:03.130Z    INFO    interval(70)    cancel_abort_after  Cancel abort_after for timer
2023-06-09T04:24:03.130Z    INFO    executor(725)   _execute_command_in_container   The return code was not 0. Sleeping for 30 seconds in case it's still writing to disk.
2023-06-09T04:24:33.143Z    INFO    interval(67)    set_abort_after_timer   Set the timer's abort_after to 660 seconds from now: abort_after = 2023-06-09 04:35:33.143353
2023-06-09T04:24:33.143Z    INFO    executor(777)   _dump_file_upload   No dump files to upload.
2023-06-09T04:24:33.144Z    INFO    interval(67)    set_abort_after_timer   Set the timer's abort_after to 180 seconds from now: abort_after = 2023-06-09 04:27:33.143992
2023-06-09T04:24:33.144Z    INFO    executor(996)   _upload_files   Allowing maximum 0.0 seconds to upload 0 bytes of files.
2023-06-09T04:24:33.144Z    INFO    interval(67)    set_abort_after_timer   Set the timer's abort_after to 60.0 seconds from now: abort_after = 2023-06-09 04:25:33.144457
2023-06-09T04:24:33.144Z    INFO    executor(737)   _execute_command_in_container   Finished _execute_command_in_container, exit code: -6

[ViktorHofer]

cc @agocke

[ViktorHofer]

Closing as I haven't seen any recent reports.