Migrate CI Workflows to Use Generic Jobs #1476
Description
We hope to have as many CI workflows across the PyTorch Ecosystem using the Generic CI Jobs. Here is the link to the documentation/migration guide to the generic CI jobs: https://github.com/pytorch/test-infra/wiki/Writing-generic-CI-jobs.
Currently we have a number of CI workflows still on CircleCI. A few other CI jobs already migrated are not completely reliable. The scope of this project is to move jobs to the generic workflows, ensure they are running reliably, and deprecate the old CircleCI workflows.
You can look through PR's in each of the domain repos and find which jobs are still using CircleCI. A brief (but not exhaustive) list here:
- Text: https://github.com/pytorch/text/pull/2031/checks?check_run_id=10864402281 (only Windows CI jobs are remaining)
- Vision: https://github.com/pytorch/vision/pull/7116/checks?check_run_id=10772480824
- TorchRec: https://github.com/pytorch/torchrec/actions/runs/3992026932/jobs/6847472773 (jobs are all using generic workflows, but most are cancelled)
- Torchaudio: No CI jobs have been migrated to GHA Generic Jobs. See CircleCI jobs here: https://github.com/pytorch/audio/pull/2933/checks?check_run_id=10489122232
- TorchData: Tests use GHA but not our generic job (and they currently use GitHub-hosted runners).
Other ecosystem repos like TorchX, TorchRL, TorchEval, TorchTNT, etc. should also be moved be to generic CI jobs over time.
Our typical process for these migrations are as follows:
- Create PR to introduce CI workflow based on generic job in the domain repo. Ensure domain POC approval before merging.
- Wait ~2 weeks and ensure the job is reliable (passing/failing when appropriate on PR's). If all is well, then deprecate the old CircleCI job (again with domain POC approval).