Enable multi agent parallel testing in CI

[majocha]

Basically I propose to revive #17765

Given that at least xUnit tests are now fully isolated and capable of parallel execution, we could speed up the CI considerably.
This can be done either using VSTest@3 task or, for portability, with just a simple script and --filtering the tests.

Major hurdle of course is this azure-pipelines-PR.yml

It would be nice to maybe split it into smaller chunks with templates, extract build and testing into separate stages. Make the testing stage download build artifacts, instead of building everything so many times.

We could look for inspiration at how https://github.com/dotnet/roslyn does it.

[abonie]

@T-Gro did some experiments using the built in VsTask, abandoned here https://github.com/dotnet/fsharp/pull/17765/files .

[majocha]

Yes exactly, that's what I was thinking about.

[T-Gro]

I am happy to assist if you are interested into pursueing this.
The VSTest@3 did work with some minor bugs - e.g. I had the feeling that parametrized tests were always part of the same bucket (= it does not know how to split them) and we do have TypeProviderTests which just take ages. But that was before your single-machine parallelization.

But if that is the only issue, I think we could do a "hack" and just split the TP test e.g. into 4 different blocks, all calling the same underlying test function eventually.

IIRC, I then got a little stuck on correctly transferring build artefacts from one machine to all the others.
Thinking about it now, maybe "who cares?" since it will not save that much time anyway - the remaining machines would be anyway staying idle and waiting for the first one, and dotnet/fsharp is neglectable when it comes to resource consumption.

i.e. starting out with the first approach, each machine just building its own compiler and using that, would be OKish?

[majocha]

I think there'll be no problem with splitting the tests.
We can assign pseudo-randomly into batches for example by hashing the name:

It could do as first approximation.

I'll try to come up with something. With some nudging Copilot seems to be quite helpful with such tasks.