Skip to content
Merged
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
841de2c
consolidate Linux workflows on CPU and GPU
pmeier Feb 8, 2023
7e90838
add sleep for ssh
pmeier Feb 8, 2023
99d6332
disable CPU for now
pmeier Feb 8, 2023
d81811d
sleep right away
pmeier Feb 8, 2023
81c45a2
Merge branch 'main' into consolidate-linux-ci
pmeier Feb 8, 2023
179f224
Merge branch 'main' into consolidate-linux-ci
pmeier Feb 8, 2023
52e7cea
Merge branch 'consolidate-linux-ci' of https://github.com/pmeier/visi…
pmeier Feb 8, 2023
c87f260
fix GPU workflow inputs
pmeier Feb 8, 2023
d7a11cb
Merge branch 'main' into consolidate-linux-ci
pmeier Feb 9, 2023
7f0ac72
Merge branch 'main' into consolidate-linux-ci
pmeier Feb 9, 2023
11231a3
Merge branch 'main' into consolidate-linux-ci
pmeier Feb 9, 2023
126c27a
Merge branch 'main' into consolidate-linux-ci
pmeier Feb 9, 2023
ea8d1f6
Merge branch 'main' into consolidate-linux-ci
pmeier Feb 20, 2023
11a2d68
refactor script
pmeier Feb 20, 2023
de913f9
cleanup
pmeier Feb 20, 2023
6931577
also install ffmpeg
pmeier Feb 20, 2023
701e71f
try ffmpeg<5
pmeier Feb 21, 2023
1c86593
try ffmpeg<4.3
pmeier Feb 21, 2023
469008e
Merge branch 'main' into consolidate-linux-ci
pmeier Feb 21, 2023
7b22e3d
Update .github/unittest.sh
pmeier Feb 21, 2023
9cc3e4d
Merge branch 'main' into consolidate-linux-ci
pmeier Feb 21, 2023
240cad6
Merge branch 'consolidate-linux-ci' of https://github.com/pmeier/visi…
pmeier Feb 21, 2023
42f37bf
Merge branch 'main' into consolidate-linux-ci
pmeier Feb 27, 2023
62dd31a
try 3.11 on linux
pmeier Feb 27, 2023
d244e1f
only try to install av on < 3.11
pmeier Feb 27, 2023
d5ea9ee
add comment for ffmpeg pin
pmeier Feb 27, 2023
2cbba69
add missing --pre
pmeier Feb 27, 2023
0849838
fix bash multiline comment
pmeier Feb 27, 2023
d0c3ec9
fix conditional
pmeier Feb 27, 2023
2819673
fix PyTorch installation
pmeier Feb 27, 2023
f567e51
Merge branch 'main' into consolidate-linux-ci
pmeier Feb 27, 2023
ab4dafe
Merge branch 'main' into consolidate-linux-ci
pmeier Mar 1, 2023
8e10b25
use defaults channel over conda-forge
pmeier Mar 1, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 80 additions & 0 deletions .github/unittest.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
#!/usr/bin/env bash

set -euo pipefail

# Prepare conda
CONDA_PATH=$(which conda)
eval "$(${CONDA_PATH} shell.bash hook)"
Comment on lines +5 to +7
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC, all the generic jobs come with conda pre-installed and use bash, right? If so, could we maybe run conda init before we enter the script? Otherwise stuff like conda activate doesn't work without running the highlighted commands before.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At least the conda part is not true for Windows. @osalpekar is this intentional?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not intentional - since this is the first application of the windows job, we'll need to identify and patch these gaps as we go :) This line would need to be added to the windows_job to ensure conda is installed: https://github.com/pytorch/test-infra/blob/main/.github/workflows/macos_job.yml#L84


echo '::group::Set PyTorch conda channel and wheel index'
# TODO: Can we maybe have this as environment variable in the job template? For example, `IS_RELEASE`.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will open an issue in test-infra for that

if [[ (${GITHUB_EVENT_NAME} = 'pull_request' && (${GITHUB_BASE_REF} = 'release'*)) || (${GITHUB_REF} = 'refs/heads/release'*) ]]; then
CHANNEL_ID=test
else
CHANNEL_ID=nightly
fi
PYTORCH_CONDA_CHANNEL=pytorch-"${CHANNEL_ID}"
echo "PYTORCH_CONDA_CHANNEL=${PYTORCH_CONDA_CHANNEL}"

case $GPU_ARCH_TYPE in
cpu)
GPU_ARCH_ID="cpu"
;;
cuda)
VERSION_WITHOUT_DOT=$(echo "${GPU_ARCH_VERSION}" | sed 's/\.//')
GPU_ARCH_ID="cu${VERSION_WITHOUT_DOT}"
;;
*)
echo "Unknown GPU_ARCH_TYPE=${GPU_ARCH_TYPE}"
exit 1
;;
esac
PYTORCH_WHEEL_INDEX="https://download.pytorch.org/whl/${CHANNEL_ID}/${GPU_ARCH_ID}"
echo "PYTORCH_WHEEL_INDEX=${PYTORCH_WHEEL_INDEX}"
echo '::endgroup::'

echo '::group::Create build environment'
conda create \
--name ci \
--quiet --yes \
python="${PYTHON_VERSION}" pip \
ninja libpng jpeg 'ffmpeg<4.3' \
-c "${PYTORCH_CONDA_CHANNEL}" \
-c conda-forge
conda activate ci
pip install --progress-bar=off --upgrade setuptools
echo '::endgroup::'

echo '::group::Install PyTorch'
# Due to the supply chain attack in Dec 2022 (https://pytorch.org/blog/compromised-nightly-dependency/), we host all
# third-party dependencies on Linux on our own indices and *don't* install them from PyPI.
case "$(uname -s)" in
Linux*)
INDEX_TYPE="index-url"
;;
*)
INDEX_TYPE="extra-index-url"
esac

pip install --progress-bar=off torch "--${INDEX_TYPE}=${PYTORCH_WHEEL_INDEX}"

if [[ $GPU_ARCH_TYPE = 'cuda' ]]; then
python3 -c "import torch; exit(not torch.cuda.is_available())"
fi
echo '::endgroup::'

echo '::group::Install TorchVision'
python setup.py develop
echo '::endgroup::'

echo '::group::Collect PyTorch environment information'
python -m torch.utils.collect_env
echo '::endgroup::'

echo '::group::Install testing utilities'
pip install --progress-bar=off pytest pytest-mock pytest-cov
echo '::endgroup::'

echo '::group::Run tests'
pytest --durations=25
echo '::endgroup::'
57 changes: 0 additions & 57 deletions .github/workflows/test-linux-cpu.yml

This file was deleted.

61 changes: 0 additions & 61 deletions .github/workflows/test-linux-gpu.yml

This file was deleted.

40 changes: 40 additions & 0 deletions .github/workflows/test-linux.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
name: Unit-tests on Linux

on:
pull_request:
push:
branches:
- nightly
- main
- release/*
workflow_dispatch:

jobs:
tests:
strategy:
matrix:
python-version:
- "3.8"
- "3.9"
- "3.10"
runner: ["linux.12xlarge"]
gpu-arch-type: ["cpu"]
include:
- python-version: 3.8
runner: linux.g5.4xlarge.nvidia.gpu
gpu-arch-type: cuda
gpu-arch-version: "11.7"
fail-fast: false
uses: pytorch/test-infra/.github/workflows/linux_job.yml@main
with:
repository: pytorch/vision
runner: ${{ matrix.runner }}
gpu-arch-type: ${{ matrix.gpu-arch-type }}
gpu-arch-version: ${{ matrix.gpu-arch-version }}
timeout: 120
script: |
export PYTHON_VERSION=${{ matrix.python-version }}
export GPU_ARCH_TYPE=${{ matrix.gpu-arch-type }}
export GPU_ARCH_VERSION=${{ matrix.gpu-arch-version }}

./.github/unittest.sh