cmd/go: checksum mismatch with private dependent module with GIT LFS files

[paaloeye]

Summary

Due to this issue, the generation of the checksums for private modules depends on the availability of git-lfs binary.

I propose to pass GIT_LFS_SKIP_SMUDGE=1 to git binary when dependent modules are being fetched.

Important conditions for the repro:

1. org/private-repo is a private repository
2. org/private-repo has GIT LFS files
3. git-lfs binary is available
4. git-lfs filter is configured (see below).

$ git config -l 
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f

What version of Go are you using (go version)?

$ go version
go version go1.14.2 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="auto"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/vscode/.cache/go-build"
GOENV="/home/vscode/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY="github.com/org/private-repo"
GONOSUMDB="github.com/org/private-repo"
GOOS="linux"
GOPATH="/go"
GOPRIVATE="github.com/org/private-repo"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/workspaces/repo/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build765684387=/tmp/go-build -gno-record-gcc-switches"

What did you do?

$ go run main.go

What did you expect to see?

The binary being run.

What did you see instead?

verifying github.com/org/private-repo@v0.1.1: checksum mismatch
        downloaded: h1:VHvAPce3Ul6tvbVTQiXQtX6mQDntkwvStfgCAzQrrAY=
        go.sum:     h1:PmwChWv1bky83vNIbS3xXnnB8iWcrMkTX2tCjtXaFI8

Workaround

$ export GIT_LFS_SKIP_SMUDGE=1

[paaloeye]

Public dependencies with Git LFS files might be affected as well.

[toothrot]

/cc @bcmills @jayconrod @matloob

[bcmills]

What did you do?

$ go run main.go

Since you have obtained two different checksums, it seems likely that the repo has been cloned using two different configurations: one when the module was fetched initially (creating the go.sum entry), and a different one when the module was re-fetched (creating the conflicting checksum).

The “What did you do?” step needs to describe both of those configurations, not just the most recent one. (Otherwise, how can we reproduce the failure to ensure that it is fixed?)

[bcmills]

The go command uses git archive under the hood.

According to git-lfs/git-lfs#1322 (comment),

If you have Git LFS enabled (i.e., the filter rules are properly set up via git lfs install), a recent version of git archive will include the LFS files in it, even in a bare repository.

So it's not obvious to me why GIT_LFS_SKIP_SMUDGE=1 would be the right resolution here: why would git-lfs users expect Go modules to include (or omit) LFS files, and under what conditions? Does this behavior vary with git and git-lfs versions, and is upgrading to a more recent git and/or git-lfs binary a viable workaround?

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)