Skip to content

[wip] fix horovod installation #12353

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 24 commits into from
Closed

[wip] fix horovod installation #12353

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
effe403
horovodrun --check-build
akihironitta Mar 17, 2022
9fa4e21
rm concurrency limit
akihironitta Mar 17, 2022
d321e1f
Reinstall horovod
akihironitta Mar 17, 2022
c979144
Extend timeout
akihironitta Mar 17, 2022
90b1b21
set back concurrency
akihironitta Mar 17, 2022
56eb554
revert docker push
akihironitta Mar 17, 2022
d1b3e15
Skip test
akihironitta Mar 17, 2022
68a13c6
Skip test
akihironitta Mar 17, 2022
a879a50
Pin docker image sha
akihironitta Mar 17, 2022
d2351ea
Fix syntax
akihironitta Mar 17, 2022
7251f79
Add comment and update sha256 in gpu-benchmark.yml
akihironitta Mar 18, 2022
947fd38
Revert "Skip test"
akihironitta Mar 18, 2022
bd9d455
Revert "Skip test"
akihironitta Mar 18, 2022
ed617fb
Merge branch 'ci/fix-cuda-horovod2' into ci/fix-cuda-horovod
akihironitta Mar 18, 2022
9ede903
Merge branch 'master' into ci/fix-cuda-horovod
akihironitta Mar 18, 2022
c6f65dd
Fail fast on horovod installation
akihironitta Mar 18, 2022
2ffbc03
Fix syntax
akihironitta Mar 18, 2022
9541715
No more syntax error please
akihironitta Mar 18, 2022
291c09c
Simplify Dockerfile for debugging
akihironitta Mar 18, 2022
de2a7f1
Dump check-build to log output
akihironitta Mar 18, 2022
c36276c
rm concurrency limit
akihironitta Mar 18, 2022
cf11dca
Comment reinstallation
akihironitta Mar 18, 2022
707b7cf
Pin horovod==0.24.2
akihironitta Mar 18, 2022
d57d637
Pin horovod==0.24.1
akihironitta Mar 18, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions .github/workflows/ci_dockers.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,10 @@ on: # Trigger the workflow on push or pull request, but only for the master bran
- ".github/workflows/events-nightly.yml"
- "setup.py"

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}-${{ github.head_ref }}
cancel-in-progress: ${{ ! (github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/heads/release/')) }}
# FIXME
# concurrency:
# group: ${{ github.workflow }}-${{ github.ref }}-${{ github.head_ref }}
# cancel-in-progress: ${{ ! (github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/heads/release/')) }}

jobs:
build-PL:
Expand All @@ -43,7 +44,7 @@ jobs:
PYTORCH_VERSION=${{ matrix.pytorch_version }}
file: dockers/release/Dockerfile
push: false
timeout-minutes: 50
timeout-minutes: 70 # FIXME

build-XLA:
runs-on: ubuntu-20.04
Expand Down
35 changes: 7 additions & 28 deletions dockers/base-cuda/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,38 +113,17 @@ RUN \
export HOROVOD_BUILD_CUDA_CC_LIST=${HOROVOD_BUILD_CUDA_CC_LIST//"."/""} && \
cat ./requirements/horovod.txt && \
cmake --version && \
pip install --no-cache-dir -r ./requirements/horovod.txt && \
pip install --no-cache-dir "horovod==0.24.1" && \
PYTHONUNBUFFERED=1 horovodrun --check-build && \
# pip uninstall -y horovod && \
# pip install --no-cache-dir -r ./requirements/horovod.txt && \
# PYTHONUNBUFFERED=1 horovodrun --check-build && \
python -c "from horovod.torch import nccl_built; nccl_built()" && \
rm -rf requirements/

RUN \
CUDA_VERSION_MAJOR=$(python -c "import torch; print(torch.version.cuda.split('.')[0])") && \
py_ver=$(python -c "print(int('$PYTHON_VERSION'.split('.') >= '3.9'.split('.')))") && \
# install DALI, needed for examples
# todo: waiting for 1.4 - https://github.com/NVIDIA/DALI/issues/3144#issuecomment-877386691
if [ $py_ver -eq "0" ]; then \
pip install --extra-index-url https://developer.download.nvidia.com/compute/redist "nvidia-dali-cuda${CUDA_VERSION_MAJOR}0>1.0" ; \
python -c 'from nvidia.dali.pipeline import Pipeline' ; \
fi

RUN \
# install NVIDIA apex
pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" https://github.com/NVIDIA/apex/archive/refs/heads/master.zip && \
python -c "from apex import amp"

RUN \
# install FairScale
pip install fairscale==0.4.0 && \
python -c "import fairscale; print(fairscale.__version__)"

RUN \
# install DeepSpeed
pip install deepspeed==0.5.7 && \
python -c "import deepspeed; print(deepspeed.__version__)"

RUN \
# Show what we have
pip --version && \
pip list && \
python -c "import sys; ver = sys.version_info ; assert f'{ver.major}.{ver.minor}' == '$PYTHON_VERSION', ver" && \
python -c "import torch; assert torch.__version__.startswith('$PYTORCH_VERSION'), torch.__version__" && \
python -c "import horovod.torch"
python -c "import torch; assert torch.__version__.startswith('$PYTORCH_VERSION'), torch.__version__"