Early EOF errors when running git fetch over ssh #1322
Description
If I configure the Git for Windows client to use the SSH version that ships with Windows, I occasionally see the following error during git fetch:
fatal: the remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed
Here's a log that includes ssh -vvv output:
fetch.log
And here's the test script that I used, which replays the same fetch 100 times in a row:
fetchloop.sh.txt
I believe that this is a bug in the SSH client that ships with Windows, because:
- It does not reproduce with the version of OpenSSH built in to Git for Windows.
- It reproduces with multiple Git server implementations that don't share Git or SSH code.
The bug also seems data-shape dependent in that it reproduces more or less frequently depending on the exact state of the Git repos on the client and server. I have snapshots of the client and server Git repo state. I can't share it publicly, can can share it internally with Microsoft employees.
My client OS is Windows 10 1809 Build 17763.253
((Get-Item (Get-Command sshd).Source).VersionInfo.FileVersion) says 7.7.2.2
The Git servers that I have tested against are:
- Ubuntu 18.04.1 LTS in a Hyper-V VM running in the same machine as my git client. I followed the instructions here: https://git-scm.com/book/en/v2/Git-on-the-Server-Setting-Up-the-Server
- Azure DevOps, which uses custom implementations of both Git and SSH which do not share code with either Git for Windows (we wrote it from scratch) or OpenSSH (ours is based on a proprietary SSH library (SecureBlackBox)). I am the current maintainer of the code that handles Git SSH requests for Azure DevOps.
From messing around with the Azure DevOps SSH code, I found the following:
- I'm fairly certain that git.exe, which shells out to ssh.exe, is hitting EOF before the end of the expected fetch response.
- I'm fairly certain our server is sending the correct SSH_MSG_CHANNEL_DATA packets for the entire fetch response
- I can "fix" the issue by adding a sleep immediately before sending SSH_MSG_CHANNEL_CLOSE from our server.
So this looks like a race condition where channel data from the server is lost when the client receives SSH_MSG_CHANNEL_CLOSE.
BTW, I tried searching for related issues and found #752 and a few others, but they all seemed to be issues with running the Windows version of sshd on the server, as opposed to the client.
@dscho this should probably block git-for-windows/git#1981