How best to reproduce CI perf results locally?

[Nadrieril]

In this PR, rust-timer found a 4% regression on instruction counts on the match-stress benchmark. When measuring locally, I consistently find instead a 14% improvement on that same benchmark (even after rebasing on master). This is pretty annoying because I can't try to find the source of the regression locally at all.

Do you know what could cause such a difference? Could a difference in architecture explain that? Is there anything I could do to make the results closer to CI? Maybe flags or environment variables I could set?

[Mark-Simulacrum]

How are you building locally? CI uses a bunch of different settings than default (e.g., PGO, different LTO configuration), all of which can have large impact, particularly for stress tests dominated by a small amount of code.

[Nadrieril]

My config.toml is just profile = "compiler" and I use the binary produced by ./x.py test tests/ui found in ./build/host/stage1/bin/rustc

[nnethercote]

When the CI results don't match my local results I usually assume that PGO is the cause. +4% vs -14% is an unusually large difference, though!

[nnethercote]

Oh, one important thing: here is the config.toml I use

changelog-seen = 2

[rust]
debuginfo-level = 1
use-lld = true
jemalloc = true

If you're on Linux, the jemalloc = true is important line, because the shipped Linux compiler uses jemalloc and that can make a big difference.

[nnethercote]

I don't think there is any more to be done, so I will close this issue. Please reopen if you disagree.

[Nadrieril]

I tried jemalloc and thin LTO and the difference is the same. I assume it's PGO then, which seems like a pain to make work.

I think what could be done is a paragraph in the README that mentions these options (is there a way to fully know what settings the CI uses btw?) and PGO for the next person who is confused