runtime: mallocgc leaks memory due to profilealloc when MemProfileRate is > 0 which is true by default

[leighhopcroft]

What version of Go are you using (go version)?

$ go version
go version go1.15.3 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="auto"
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/go/pkg/mod"
GONOPROXY="gitlab.corp.wabtec.com"
GONOSUMDB="gitlab.corp.wabtec.com"
GOOS="linux"
GOPATH="/go"
GOPRIVATE="gitlab.corp.wabtec.com"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/workspaces/rm-edge-hst/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build132198306=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I reviewed the runtime source as I was looking for a stubborn memory leak in a long running process and discovered a code path that I think leaks memory. @josharian Encouraged me to raise an issue here after discussion on gophers slack #performance channel.

What did you expect to see?

No indefinite repeated calls to runtime.persistentalloc in a long running process.

What did you see instead?

1. runtime.MemProfileRate int = 512 * 1024 by default i.e. > 0 (https://golang.org/src/runtime/mprof.go?h=MemProfileRate#L493)
2. mallocgc will therefore call profilealloc periodically due to MemProfileRate > 0 (https://golang.org/src/runtime/malloc.go?h=profilealloc#L1140)
3. profilealloc will call mProf_Malloc (https://golang.org/src/runtime/malloc.go?h=mProf_Malloc#L1236)
4. mProf_Malloc will call stkbucket (https://golang.org/src/runtime/mprof.go?h=stkbucket#L344)
5. stkbucket will call newBucket (https://golang.org/src/runtime/mprof.go?h=newBucket#L240)
6. newBucket will call persistentalloc with size equal to unsafe.Sizeof(bucket{}) + uintptr(nstk)*unsafe.Sizeof(uintptr(0)) + unsafe.Sizeof(memRecord{}) (https://golang.org/src/runtime/mprof.go?h=persistentalloc#L173)

So, even if nstk == 0 the minimum persistent allocation would be unsafe.Sizeof(bucket{}) + unsafe.Sizeof(memRecord{}) which is 144 bytes (https://play.golang.org/p/5JCqMHh_2aO). If I have understood this correctly, this means that we are guaranteed to leak at least 144 bytes every MemProfileRate by default.

[leighhopcroft]

I believe that this manifests as an ever increasing MemStats.BuckHashSys value.

[randall77]

stkbucket only calls newBucket if there isn't a bucket of that type already. So there's only an allocation once for every call stack that allocates. Once that happens, there will be no more allocation.
Could be a lot of allocation, sure. But we need an allocation for every entry in the map.

[josharian]

cc @mknyszek @prattmic @aclements

[mknyszek]

As @randall77 said I'm pretty sure this is working as in intended. There are a number of places in the runtime where we intentionally leak a bounded amount of memory to simplify the implementation. In this case, the bound is proportional to the number of unique allocation points (including callers). In practice there aren't too many of them, and the rare ones virtually never show up at all. Has the additional memory use been a problem for you specifically @leighhopcroft?

[leighhopcroft]

#@randall77 I see, you are right. I must have missed that early return on line 231. I noticed that after adding some println logs to the persistentalloc I was seeing it consistently called over a 5 hour test session for an app I'm developing. I'm not sure whether it actually made a noticeable difference to RSS though as it was masked by a leak caused in a bug in my own code (incidentally the leak I was looking for when I found this). I think I may have this fixed now though so can run some further tests tomorrow to see at what point persistentalloc stops being called.

[randall77]

If you're getting a lot of buckets allocated it might be interesting to see what the stacks are and why they are not identical. There may be some trivial difference that could be ignored.

[leighhopcroft]

This was an analysis of persistentalloc calls I did regarding size. Not all are due to this specific code path but later testing showed the majority after around 50s in to the test are.

[leighhopcroft]

@randall77 does the bucket hash function map similar stacks closer together in hash space? I have added some logging to stkbucket but the output is a bit unwieldy. I was thinking if similar stacks have similar hashes it will be enough just to plot the hash value distribution and see where there might be clumps of similar values.

[randall77]

does the bucket hash function map similar stacks closer together in hash space?

No, the hash function is intended to distribute as randomly as possible.

[bradfitz]

I was talking to @rsc about this recently and we discussed perhaps changing the MemProfileRate default to 0 when runtime/pprof isn't imported.

Do you see any downside to that, @randall77?

/cc @crawshaw @josharian