build with PGO on macOS arm

[davidhewitt]

After #678 we will have PGO optimized builds for x86_64 ubuntu and windows.

We ideally will want to ship these for arm macOS also, however github actions does not have public arm macOS runners yet, so we can't run the profiling to gather PGO input.

Either we need to wait for public runners to be available, or run our own arm macOS self-hosted runner.

Possible resources found from a quick browse:
https://blog.pantsbuild.org/m1-support-self-hosted/
https://github.com/philips-labs/terraform-aws-github-runner
https://hackernoon.com/how-using-self-hosted-github-runners-can-save-you-a-fortune

Selected Assignee: @adriangb

[ellie]

Hey! I'm actually trying sort out ARM mac CI for Atuin at the moment. I spoke with @samuelcolvin on Twitter, and agreed I could sort you out too (I've ran mac CI for work in the past)

We discussed something like $200/mo for a 8gb RAM mac mini running ephemeral VMs, with one machine capable of running 2 VMs at a time. Apple licensing does not allow any further concurrency than this.

It'll be a cloud mac mini, running OCI images on tart or similar

As far as I understand, you'll be running around 100 jobs per day?

Edit: While digging through docs I just found this, which may be more suitable to your needs: https://github.com/apps/cirrus-runners

[samuelcolvin]

Thanks @ellie, sounds like we should try cirrus-runners first, and if they don't work, use your initial proposed solution.

The only advantage of your initial solution I can see is that (as I understand it?) atuin and pydantic-core could share a VM, whereas I assume cirrus wouldn't allow that.

[ellie]

The only advantage of your initial solution I can see is that (as I understand it?) atuin and pydantic-core could share a VM, whereas I assume cirrus wouldn't allow that.

Unfortunately not - I've just sorted out an old mac mini I found refurbished for very cheap, which is now running under my desk 😂 It's almost certainly not reliable enough for your needs, though I do very much appreciate your offer to work together on this 🙏

[zamazan4ik]

We ideally will want to ship these for arm macOS also, however github actions does not have public arm macOS runners yet, so we can't run the profiling to gather PGO input.

Hmm, before resolving the issue with macOS native runners I guess you can try to use the profile from x86_64 Ubuntu. I guess most of the "hot" code would be compatible between these two operating systems. If you are using the same compiler for Ubuntu and macOS - it should work.

[davidhewitt]

I understand it's advised to use exactly the same rust flags for the PGO instrumentation build (which presumably includes OS). I'm not convinced it's a good idea to port PGO profiles from ubuntu, both for the CI dependencies and also unclear impact on results.

[zamazan4ik]

and also unclear impact on results.

We can try to port PGO profiles and run benchmarks to check it :) So if this doesn't work - okay, we will wait for macOS-collected profiles. If works - great.

[davidhewitt]

https://github.blog/2023-10-02-introducing-the-new-apple-silicon-powered-m1-macos-larger-runner-for-github-actions/

[zamazan4ik]

Hmm, before resolving the issue with macOS native runners I guess you can try to use the profile from x86_64 Ubuntu. I guess most of the "hot" code would be compatible between these two operating systems. If you are using the same compiler for Ubuntu and macOS - it should work.

I was wrong - PGO profiles for some reason do not work in this way. This issue is a bit discussed in the LLVM Discord here.

[samuelcolvin]

This was fixed in #1063, sorry I forgot to close this.