Skip to content

Added cache for incremental build. #1184

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Apr 26, 2021
Merged

Added cache for incremental build. #1184

Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
48150a4
Added cache for incremental build.
buck-bot Oct 12, 2020
bc77560
Test cache
buck-bot Oct 12, 2020
b72ae9c
Added more content to cache
buck-bot Oct 12, 2020
e22ce3e
Added more content to cache
buck-bot Oct 12, 2020
668b457
Test build cache
buck-bot Oct 12, 2020
08590b4
Merge branch 'master' into cache_sphinx
brianjo Oct 12, 2020
7e4c684
Merge branch 'master' into cache_sphinx
brianjo Nov 18, 2020
ab9f90e
Merge branch 'master' into cache_sphinx
brianjo Nov 30, 2020
17fb8da
Merge branch 'master' into cache_sphinx
holly1238 Apr 26, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,8 @@ pytorch_tutorial_build_defaults: &pytorch_tutorial_build_defaults
export AWS_ACCESS_KEY_ID=${CIRCLECI_AWS_ACCESS_KEY_FOR_ECR_READ_ONLY}
export AWS_SECRET_ACCESS_KEY=${CIRCLECI_AWS_SECRET_KEY_FOR_ECR_READ_ONLY}
eval $(aws ecr get-login --region us-east-1 --no-include-email)
- restore_cache:
key: v1.0-tutorial-{{ .Environment.CIRCLE_JOB }}
- run:
name: Build
no_output_timeout: "20h"
Expand Down Expand Up @@ -164,11 +166,35 @@ pytorch_tutorial_build_defaults: &pytorch_tutorial_build_defaults
fi
set -x

# This also copies the cached build to docker.
docker cp /home/circleci/project/. "$id:/var/lib/jenkins/workspace"

export COMMAND='((echo "source ./workspace/env" && echo "sudo chown -R jenkins workspace && cd workspace && ./ci_build_script.sh") | docker exec -u jenkins -i "$id" bash) 2>&1'
echo ${COMMAND} > ./command.sh && unbuffer bash ./command.sh | ts

# Copy the last build from docker
docker cp "$id:/var/lib/jenkins/workspace/_build" /home/circleci/project
docker cp "$id:/var/lib/jenkins/workspace/docs" /home/circleci/project
docker cp "$id:/var/lib/jenkins/workspace/advanced" /home/circleci/project
docker cp "$id:/var/lib/jenkins/workspace/beginner" /home/circleci/project
docker cp "$id:/var/lib/jenkins/workspace/intermediate" /home/circleci/project
docker cp "$id:/var/lib/jenkins/workspace/prototype" /home/circleci/project
docker cp "$id:/var/lib/jenkins/workspace/recipes" /home/circleci/project
docker cp "$id:/var/lib/jenkins/workspace/src" /home/circleci/project

- save_cache:
# Save to cache for incremental build
key: v1.0-tutorial-{{ .Environment.CIRCLE_JOB }}
paths:
- /home/circleci/project/_build
- /home/circleci/project/docs
- /home/circleci/project/advanced
- /home/circleci/project/beginner
- /home/circleci/project/intermediate
- /home/circleci/project/prototype
- /home/circleci/project/recipes
- /home/circleci/project/src

pytorch_tutorial_build_worker_defaults: &pytorch_tutorial_build_worker_defaults
environment:
DOCKER_IMAGE: "308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda10.2-cudnn7-py3-gcc7"
Expand Down