Skip to content

CI: Combine conda and full testing into a single workflow #14387

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 60 commits into from
Sep 30, 2022
Merged

CI: Combine conda and full testing into a single workflow #14387

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
ebec339
Remove conda job
akihironitta Aug 25, 2022
2a97219
Remove conda job from readme
akihironitta Aug 25, 2022
f8be7ad
Remove conda jobs from checkgroup
akihironitta Aug 25, 2022
d62c7cb
Remove conda from docker builds
akihironitta Aug 25, 2022
18c7e64
Remove base-conda dockerfile
akihironitta Aug 25, 2022
6c29b6d
Rewrite the strategy matrix while keeping equivalent
akihironitta Aug 25, 2022
0f4c807
Run the workflow on this branch
akihironitta Aug 25, 2022
2dfa774
Revert "Rewrite the strategy matrix while keeping equivalent"
akihironitta Aug 25, 2022
bbf3eb7
Add PyTorch versions
akihironitta Aug 25, 2022
eb5dc5e
Run on draft and disable unrelated costly CI
akihironitta Aug 25, 2022
3926002
Revert "Run the workflow on this branch"
akihironitta Aug 25, 2022
9404680
tmp: Lightweight relevant CI
akihironitta Aug 25, 2022
f7bc4ca
Fix CI pathfilter
akihironitta Aug 25, 2022
4ed163d
Update matrix
akihironitta Sep 6, 2022
56a4474
Drop skipping logic
akihironitta Sep 6, 2022
6c6b8d7
Merge branch 'ci/drop-changed-files-macos' into ci/combine-conda-full
akihironitta Sep 7, 2022
663bd2c
pip list
akihironitta Sep 7, 2022
ab9bc82
reorder pip list
akihironitta Sep 7, 2022
b0b438e
tmp: lightweight ci
akihironitta Sep 7, 2022
76dcd71
Merge branch 'master' into ci/combine-conda-full
akihironitta Sep 7, 2022
c22b96d
Install specified pytorch
akihironitta Sep 7, 2022
c582ea2
Fix torch installation
akihironitta Sep 8, 2022
79c07c3
Uncomment steps
akihironitta Sep 8, 2022
1099a1b
Increase timeout
akihironitta Sep 8, 2022
978e69b
bad merge
akihironitta Sep 8, 2022
e5b2924
Merge branch 'master' into ci/combine-conda-full
akihironitta Sep 8, 2022
ffc5154
Revert "Run on draft and disable unrelated costly CI"
akihironitta Sep 8, 2022
2e884cc
Merge branch 'master' into ci/combine-conda-full
akihironitta Sep 8, 2022
1076c75
Update checkgroup
akihironitta Sep 8, 2022
8697e01
Update docs and remove Python/PyTorch versions
akihironitta Sep 8, 2022
622cb19
Remove pip-list
akihironitta Sep 9, 2022
41f597b
Fail if wrong pytorch version installed
akihironitta Sep 9, 2022
3fed54c
Merge branch 'master' into ci/combine-conda-full
Borda Sep 9, 2022
d54c228
Merge branch 'master' into ci/combine-conda-full
akihironitta Sep 10, 2022
1e1b97c
Merge branch 'master' into ci/combine-conda-full
akihironitta Sep 12, 2022
932eb63
Merge branch 'master' into ci/combine-conda-full
akihironitta Sep 18, 2022
93108e4
Add Python 3.8, PyTorch 1.9 job
akihironitta Sep 18, 2022
fef8842
tmp: remove azure jobs
akihironitta Sep 18, 2022
2679278
tmp: remove dockers
akihironitta Sep 18, 2022
d20b514
tmp: remove others
akihironitta Sep 18, 2022
66c285d
Run all combinations
akihironitta Sep 18, 2022
371a594
Include oldest
akihironitta Sep 18, 2022
e2df964
Exclude no Python 3.10 distributions
akihironitta Sep 18, 2022
4a7978d
tmp: no concurrency
akihironitta Sep 18, 2022
3581c97
tmp: double timeout
akihironitta Sep 18, 2022
c6e15a4
Add pytest log reporter
akihironitta Sep 18, 2022
5332ef5
Add pytest-reportlog
akihironitta Sep 18, 2022
598f315
Fewer jobs
akihironitta Sep 18, 2022
eb9d490
Revert "tmp: no concurrency"
akihironitta Sep 18, 2022
9f6ff15
fix artifact name
akihironitta Sep 18, 2022
c72c970
Merge branch 'master' into ci/combine-conda-full
akihironitta Sep 29, 2022
3556f64
Revert test reports
akihironitta Sep 29, 2022
f3f8c8c
Revert unrelated changes
akihironitta Sep 29, 2022
c7cdb7c
Revert unrelated changes
akihironitta Sep 29, 2022
114a4c0
Add the combination of ex-conda jobs
akihironitta Sep 29, 2022
cbda120
Update checkgroup
akihironitta Sep 29, 2022
3e1b712
Merge branch 'master' into ci/combine-conda-full
akihironitta Sep 29, 2022
7d7e84e
revert timeout
akihironitta Sep 29, 2022
c66ab14
remove conda job
akihironitta Sep 29, 2022
e79855f
revert docker build workflow file
akihironitta Sep 29, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 51 additions & 44 deletions .github/checkgroup.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,19 +31,23 @@ subprojects:
- "setup.cfg" # includes pytest config
checks:
# Note: updates here should be applied to the lightning_lite group
- "pl-conda (3.8, 1.10)"
- "pl-conda (3.8, 1.9)"
- "pl-conda (3.9, 1.11)"
- "pl-conda (3.9, 1.12)"
- "pl-cpu (macOS-11, 3.10, latest, stable)"
- "pl-cpu (macOS-11, 3.7, latest, stable)"
- "pl-cpu (macOS-11, 3.7, oldest, stable)"
- "pl-cpu (ubuntu-20.04, 3.10, latest, stable)"
- "pl-cpu (ubuntu-20.04, 3.7, latest, stable)"
- "pl-cpu (ubuntu-20.04, 3.7, oldest, stable)"
- "pl-cpu (windows-2022, 3.10, latest, stable)"
- "pl-cpu (windows-2022, 3.7, latest, stable)"
- "pl-cpu (windows-2022, 3.7, oldest, stable)"
- "pl-cpu (ubuntu-20.04 3.7, 1.9, oldest)"
- "pl-cpu (ubuntu-20.04 3.7, 1.12)"
- "pl-cpu (ubuntu-20.04 3.8, 1.9)"
- "pl-cpu (ubuntu-20.04 3.8, 1.10)"
- "pl-cpu (ubuntu-20.04 3.9, 1.11)"
- "pl-cpu (ubuntu-20.04 3.9, 1.12)"
- "pl-cpu (ubuntu-20.04 3.10, 1.12)"
- "pl-cpu (macos-11, 3.7, 1.9, oldest)"
- "pl-cpu (macos-11, 3.7, 1.12)"
- "pl-cpu (macos-11, 3.8, 1.10)"
- "pl-cpu (macos-11, 3.9, 1.11)"
- "pl-cpu (macos-11, 3.10, 1.12)"
- "pl-cpu (windows-2022, 3.7, 1.9, oldest)"
- "pl-cpu (windows-2022, 3.7, 1.12)"
- "pl-cpu (windows-2022, 3.8, 1.10)"
- "pl-cpu (windows-2022, 3.9, 1.11)"
- "pl-cpu (windows-2022, 3.10, 1.12)"
- "make-doctest (pytorch)"
- "make-html (pytorch)"
- "mypy"
Expand All @@ -56,28 +60,27 @@ subprojects:
# TODO: since this job cannot run on forks, it cannot be required or it will block all PL PRs from forks
#- "test-on-tpus"

- id: "pytorch_lightning: Conda"
paths:
- ".github/workflows/ci-pytorch-test-conda.yml"
checks:
- "pl-conda (3.8, 1.10)"
- "pl-conda (3.8, 1.9)"
- "pl-conda (3.9, 1.11)"
- "pl-conda (3.9, 1.12)"

- id: "pytorch_lightning: CPU"
paths:
- ".github/workflows/ci-pytorch-test-full.yml"
checks:
- "pl-cpu (macOS-11, 3.10, latest, stable)"
- "pl-cpu (macOS-11, 3.7, latest, stable)"
- "pl-cpu (macOS-11, 3.7, oldest, stable)"
- "pl-cpu (ubuntu-20.04, 3.10, latest, stable)"
- "pl-cpu (ubuntu-20.04, 3.7, latest, stable)"
- "pl-cpu (ubuntu-20.04, 3.7, oldest, stable)"
- "pl-cpu (windows-2022, 3.10, latest, stable)"
- "pl-cpu (windows-2022, 3.7, latest, stable)"
- "pl-cpu (windows-2022, 3.7, oldest, stable)"
- "pl-cpu (ubuntu-20.04 3.7, 1.9, oldest)"
- "pl-cpu (ubuntu-20.04 3.7, 1.12)"
- "pl-cpu (ubuntu-20.04 3.8, 1.9)"
- "pl-cpu (ubuntu-20.04 3.8, 1.10)"
- "pl-cpu (ubuntu-20.04 3.9, 1.11)"
- "pl-cpu (ubuntu-20.04 3.9, 1.12)"
- "pl-cpu (ubuntu-20.04 3.10, 1.12)"
- "pl-cpu (macos-11, 3.7, 1.9, oldest)"
- "pl-cpu (macos-11, 3.7, 1.12)"
- "pl-cpu (macos-11, 3.8, 1.10)"
- "pl-cpu (macos-11, 3.9, 1.11)"
- "pl-cpu (macos-11, 3.10, 1.12)"
- "pl-cpu (windows-2022, 3.7, 1.9, oldest)"
- "pl-cpu (windows-2022, 3.7, 1.12)"
- "pl-cpu (windows-2022, 3.8, 1.10)"
- "pl-cpu (windows-2022, 3.9, 1.11)"
- "pl-cpu (windows-2022, 3.10, 1.12)"

- id: "pytorch_lightning: Slow"
paths:
Expand Down Expand Up @@ -127,10 +130,6 @@ subprojects:
- ".github/workflows/*docker*.yml"
- "setup.py"
checks:
- "build-conda (3.8, 1.9, 11.1.1)"
- "build-conda (3.8, 1.10.1, 11.1.1)"
- "build-conda (3.9, 1.11, 11.3.1)"
- "build-conda (3.9, 1.12, 11.3.1)"
- "build-cuda (3.8, 1.9, 11.1.1)"
- "build-cuda (3.9, 1.10, 11.3.1)"
- "build-cuda (3.9, 1.11, 11.3.1)"
Expand Down Expand Up @@ -166,15 +165,23 @@ subprojects:
- "lightning-lite (GPUs)"
- "mypy"
# Lite also requires PL checks as it depends on Lite
- "pl-cpu (macOS-11, 3.10, latest, stable)"
- "pl-cpu (macOS-11, 3.7, latest, stable)"
- "pl-cpu (macOS-11, 3.7, oldest, stable)"
- "pl-cpu (ubuntu-20.04, 3.10, latest, stable)"
- "pl-cpu (ubuntu-20.04, 3.7, latest, stable)"
- "pl-cpu (ubuntu-20.04, 3.7, oldest, stable)"
- "pl-cpu (windows-2022, 3.10, latest, stable)"
- "pl-cpu (windows-2022, 3.7, latest, stable)"
- "pl-cpu (windows-2022, 3.7, oldest, stable)"
- "pl-cpu (ubuntu-20.04 3.7, 1.9, oldest)"
- "pl-cpu (ubuntu-20.04 3.7, 1.12)"
- "pl-cpu (ubuntu-20.04 3.8, 1.9)"
- "pl-cpu (ubuntu-20.04 3.8, 1.10)"
- "pl-cpu (ubuntu-20.04 3.9, 1.11)"
- "pl-cpu (ubuntu-20.04 3.9, 1.12)"
- "pl-cpu (ubuntu-20.04 3.10, 1.12)"
- "pl-cpu (macos-11, 3.7, 1.9, oldest)"
- "pl-cpu (macos-11, 3.7, 1.12)"
- "pl-cpu (macos-11, 3.8, 1.10)"
- "pl-cpu (macos-11, 3.9, 1.11)"
- "pl-cpu (macos-11, 3.10, 1.12)"
- "pl-cpu (windows-2022, 3.7, 1.9, oldest)"
- "pl-cpu (windows-2022, 3.7, 1.12)"
- "pl-cpu (windows-2022, 3.8, 1.10)"
- "pl-cpu (windows-2022, 3.9, 1.11)"
- "pl-cpu (windows-2022, 3.10, 1.12)"
- "make-doctest (pytorch)"
- "make-html (pytorch)"
- "pytorch-lightning (GPUs)"
Expand Down
22 changes: 12 additions & 10 deletions .github/workflows/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,23 +4,25 @@

## Unit and Integration Testing

| workflow name | workflow file | action | accelerator\* | (Python, PyTorch) | OS |
| -------------------------- | ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | ------------------------------------------------- | ------------------- |
| Test PyTorch full | .github/workflows/ci-pytorch-test-full.yml | Run all tests except for accelerator-specific, standalone and slow tests. | CPU | (3.7, 1.9), (3.7, 1.12), (3.9, 1.9), (3.9, 1.12) | linux, mac, windows |
| Test PyTorch with Conda | .github/workflows/ci-pytorch-test-conda.yml | Same as ci-pytorch-test-full.yml but with dependencies installed with conda. | CPU | (3.8, 1.9), (3.8, 1.10), (3.8, 1.11), (3.9, 1.12) | linux |
| Test slow | .github/workflows/ci-pytorch-test-slow.yml | Run only slow tests. Slow tests usually need to spawn threads and cannot be speed up or simplified. | CPU | (3.7, 1.11) | linux, mac, windows |
| pytorch-lightning (IPUs) | .azure-pipelines/ipu-tests.yml | Run only IPU-specific tests. | IPU | (3.8, 1.9) | linux |
| pytorch-lightning (HPUs) | .azure-pipelines/hpu-tests.yml | Run only HPU-specific tests. | HPU | (3.8, 1.10) | linux |
| pytorch-lightning (GPUs) | .azure-pipelines/gpu-tests.yml | Run all CPU and GPU-specific tests, standalone, and examples. Each standalone test needs to be run in separate processes to avoid unwanted interactions between test cases. | GPU | (3.9, 1.12) | linux |
| PyTorchLightning.Benchmark | .azure-pipelines/gpu-benchmark.yml | Run speed/memory benchmarks for parity with pure PyTorch. | GPU | (3.9, 1.12) | linux |
| test-on-tpus | .circleci/config.yml | Run only TPU-specific tests. | TPU | (3.7, 1.12) | linux |
| workflow name | workflow file | action | accelerator\* |
| -------------------------- | ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- |
| Test PyTorch full | .github/workflows/ci-pytorch-test-full.yml | Run all tests except for accelerator-specific, standalone and slow tests. | CPU |
| Test PyTorch slow | .github/workflows/ci-pytorch-test-slow.yml | Run only slow tests. Slow tests usually need to spawn threads and cannot be speed up or simplified. | CPU |
| pytorch-lightning (IPUs) | .azure-pipelines/ipu-tests.yml | Run only IPU-specific tests. | IPU |
| pytorch-lightning (HPUs) | .azure-pipelines/hpu-tests.yml | Run only HPU-specific tests. | HPU |
| pytorch-lightning (GPUs) | .azure-pipelines/gpu-tests.yml | Run all CPU and GPU-specific tests, standalone, and examples. Each standalone test needs to be run in separate processes to avoid unwanted interactions between test cases. | GPU |
| PyTorchLightning.Benchmark | .azure-pipelines/gpu-benchmark.yml | Run speed/memory benchmarks for parity with pure PyTorch. | GPU |
| test-on-tpus | .circleci/config.yml | Run only TPU-specific tests. | TPU |

- \*Accelerators used in CI

- GPU: 2 x NVIDIA Tesla V100
- TPU: Google GKE TPUv3
- IPU: [Colossus MK1 IPU](https://www.graphcore.ai/products/ipu)
- HPU: [Intel Habana Gaudi SYS-420GH-TNGR](https://www.supermicro.com/en/products/system/AI/4U/SYS-420GH-TNGR) which has 8 Gaudi accelerators

- To check which versions of Python or PyTorch are used for testing in our CI, see the corresponding workflow files or checkgroup cofig file at [`.github/checkgroup.yml`](../checkgroup.yml).

## Documentation

| workflow file | action |
Expand Down
38 changes: 0 additions & 38 deletions .github/workflows/ci-pytorch-dockers.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,44 +127,6 @@ jobs:
env:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

build-conda:
runs-on: ubuntu-20.04
strategy:
fail-fast: false
matrix:
include:
- {python_version: "3.8", pytorch_version: "1.9", cuda_version: "11.1.1"}
- {python_version: "3.8", pytorch_version: "1.10.1", cuda_version: "11.1.1"}
- {python_version: "3.9", pytorch_version: "1.11", cuda_version: "11.3.1"}
- {python_version: "3.9", pytorch_version: "1.12", cuda_version: "11.3.1"}
steps:
- uses: actions/checkout@v3
- uses: docker/setup-buildx-action@v2
- uses: docker/login-action@v2
if: env.PUSH_TO_HUB == 'true'
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- uses: docker/build-push-action@v3
with:
build-args: |
PYTHON_VERSION=${{ matrix.python_version }}
PYTORCH_VERSION=${{ matrix.pytorch_version }}
CUDA_VERSION=${{ matrix.cuda_version }}
file: dockers/base-conda/Dockerfile
push: ${{ env.PUSH_TO_HUB }}
tags: pytorchlightning/pytorch_lightning:base-conda-py${{ matrix.python_version }}-torch${{ matrix.pytorch_version }}
timeout-minutes: 95
- uses: ravsamhq/notify-slack-action@v2
if: failure() && env.PUSH_TO_HUB == 'true'
with:
status: ${{ job.status }}
token: ${{ secrets.GITHUB_TOKEN }}
notification_title: ${{ format('Conda; {0} py{1} for *{2}*', runner.os, matrix.python_version, matrix.pytorch_version) }}
message_format: '{emoji} *{workflow}* {status_message}, see <{run_url}|detail>, cc: <@U01A5T7EY9M>' # akihironitta
env:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

build-ipu:
runs-on: ubuntu-20.04
strategy:
Expand Down
110 changes: 0 additions & 110 deletions .github/workflows/ci-pytorch-test-conda.yml
View file
Open in desktop

This file was deleted.

Loading