Skip to content

[SPARK-32695][INFRA] Explicitly cache and hash 'build' directly in GitHub Actions #29536

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 7 commits into from
Closed

[SPARK-32695][INFRA] Explicitly cache and hash 'build' directly in GitHub Actions #29536

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
088bfbb
Add 'build' and 'project/build.properties' into cache key of SBT and …
HyukjinKwon Aug 25, 2020
cfe5e92
Changes in build
HyukjinKwon Aug 25, 2020
6f78677
Revert "Changes in build"
HyukjinKwon Aug 25, 2020
63032da
Whitelist everything
HyukjinKwon Aug 25, 2020
1f4706e
Changes in build
HyukjinKwon Aug 25, 2020
3900892
Revert "Changes in build"
HyukjinKwon Aug 25, 2020
42d4e5c
Cleaner
HyukjinKwon Aug 25, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions .github/workflows/build_and_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,10 +99,14 @@ jobs:
run: git merge --progress --ff-only origin/${{ github.event.inputs.target }}
# Cache local repositories. Note that GitHub Actions cache has a 2G limit.
- name: Cache Scala, SBT, Maven and Zinc
uses: actions/cache@v1
uses: actions/cache@v2
Copy link
Member

@gengliangwang gengliangwang Aug 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is the difference between v1 and v2?

Copy link
Member Author

@HyukjinKwon HyukjinKwon Aug 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In order to allow multiple file path (https://github.com/actions/cache#whats-new). It is also to make it consistent in this file. cache@v2 is used in all places at this file except here.

Copy link
Member Author

@HyukjinKwon HyukjinKwon Aug 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previously, other files like build/mvn and build/sbt are also cached and overwritten. So, when you have some changes there, they are ignored.

with:
path: build
key: build-${{ hashFiles('**/pom.xml') }}
path: |
build/apache-maven-*
build/zinc-*
build/scala-*
build/*.jar
key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
Copy link
Member

@maropu maropu Aug 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Ah, I see. glob patterns like build/*' cannot be used in hashFilesnow...)

restore-keys: |
build-
- name: Cache Maven local repository
Expand All @@ -116,7 +120,7 @@ jobs:
uses: actions/cache@v2
with:
path: ~/.ivy2/cache
key: ${{ matrix.java }}-${{ matrix.hadoop }}-ivy-${{ hashFiles('**/pom.xml') }}-${{ hashFiles('**/plugins.sbt') }}
key: ${{ matrix.java }}-${{ matrix.hadoop }}-ivy-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
${{ matrix.java }}-${{ matrix.hadoop }}-ivy-
- name: Install JDK ${{ matrix.java }}
Expand Down