Add performance regression tests and benchmarks

[cocreature]

It would be good to have some performance regression tests. We have a small wrapper around lsp-tests in DAML https://github.com/digital-asset/daml/blob/master/compiler/damlc/ide-debug-driver/README.md that makes it easy to script sessions which I’ve used successfully in the past to track down performance issues which might come in handy here.

[domenkozar]

Refs https://github.com/haskell/ghcide/issues/298 (#1007)

[ndmitchell]

Would be good to include benchmarks, which might be things that run for infeasibly long to include in tests, but are great at measuring and improving things in the region of 5% or less.

[ndmitchell]

Copying over @pepeiborra's comment:

How to prove that they are capturing real workloads? Perhaps use haskell-lsp to record and replay real interactions with vs code, emacs, rather than modelling them with lsp-test

I looked at this a bit. I think testing on real world workloads may be very important, since otherwise we can't capture all the nuances of what is being watched via notifications etc.

[pepeiborra]

@mpardalos - do you have plans to do this over the summer? It’s sorely needed!

[mpardalos]

@pepeiborra Yes! I am planning on recording some sessions for this reason.

I was planning on using lsp-test to run those sessions. I spoke with my GSoC mentor about this (@mpickering) and he told me that apparently lsp-test currently has problems with replaying sessions (it doesn't respect request timings) which might be a blocker for now? I couldn't find the relevant issue there so he probably has a better idea of what the problem is.

[pepeiborra]

I'm working on a simple benchmark based on https://neilmitchell.blogspot.com/2020/05/fixing-space-leaks-in-ghcide.html

[jneira]

@cocreature the project already has a benchmark, comparing branches and running in ci, courtesy of @pepeiborra, so only regression tests would be needed to close this?

[cocreature]

@jneira yeah I think so.

[pepeiborra]

Detecting regressions in allocations would be sweet. Steps to do this:

1. Enhance the benchmark outputs to include the total allocated bytes
2. Add a step to the bench CI script to check that the allocated bytes did not go up by more than a certain threshold

[pepeiborra]

1 is done, but 2 is still very much needed