filter whiteout files with `find` (instead of using `grep`) in `create_tarball.sh` #21

bedroge · 2025-06-24T10:39:48Z

This should solve the issues in the tarball creation steps that we see in for instance EESSI/software-layer#1131. There are some emtpy directories in the init folder, and then the grep returns 1. By adding || true (we already did that for a few other grep commands) it will always return 0.

edit: instead of adding || true to the grep command, we decided to add an option to the find command for excluding whiteout files.

trz42

Looks good.

An alternative way might be to use plain find with an additional condition, e.g.,

find /path/to/upper -type f ! -name '.wh.*'

Anyhow, up to you to decide if that's worth another change or we go with your change.

bedroge · 2025-06-24T12:00:22Z

Looks good.

An alternative way might be to use plain find with an additional condition, e.g.,
find /path/to/upper -type f ! -name '.wh.*'
Anyhow, up to you to decide if that's worth another change or we go with your change.

No strong preference, but I changed the find commands.

bedroge · 2025-06-24T12:03:54Z

Tested this version interactively in a bot/inspect.sh session for PR EESSI/software-layer#1131:

 ./create_tarball.sh /tmp 2023.06 x86_64/amd/zen2 ""  /eessi_bot_job/eessi-2023.06-software-linux-x86_64-amd-zen2-17507550540.tar.gz
>> tmpdir: /tmp/tmp.fecT0K6YhQ
>> Collecting list of files/directories to include in tarball via /tmp/software.eessi.io/overlay-upper/versions...
handling Bazel/6.1.0-GCCcore-12.3.0
handling Java/11.0.27
handling ml_dtypes/.wh..wh..opq
handling ml_dtypes/0.3.2-gfbf-2023a
handling tensorboard/.wh..wh..opq
handling tensorboard/2.15.1-gfbf-2023a
wrote file list to /tmp/tmp.fecT0K6YhQ/files.list.txt
2023.06/software/linux/x86_64/amd/zen2/modules/all/Java/11.0.27.lua
2023.06/software/linux/x86_64/amd/zen2/modules/all/Bazel/6.1.0-GCCcore-12.3.0.lua
2023.06/software/linux/x86_64/amd/zen2/modules/all/ml_dtypes/0.3.2-gfbf-2023a.lua
2023.06/software/linux/x86_64/amd/zen2/modules/all/tensorboard/2.15.1-gfbf-2023a.lua
2023.06/software/linux/x86_64/amd/zen2/modules/lang/Java/11.0.27.lua
2023.06/software/linux/x86_64/amd/zen2/modules/devel/Bazel/6.1.0-GCCcore-12.3.0.lua
2023.06/software/linux/x86_64/amd/zen2/modules/tools/ml_dtypes/0.3.2-gfbf-2023a.lua
2023.06/software/linux/x86_64/amd/zen2/modules/lib/tensorboard/2.15.1-gfbf-2023a.lua
2023.06/software/linux/x86_64/amd/zen2/software/Bazel/6.1.0-GCCcore-12.3.0
2023.06/software/linux/x86_64/amd/zen2/software/Java/11.0.27
2023.06/software/linux/x86_64/amd/zen2/software/ml_dtypes/0.3.2-gfbf-2023a
2023.06/software/linux/x86_64/amd/zen2/software/tensorboard/2.15.1-gfbf-2023a
wrote module file list to /tmp/tmp.fecT0K6YhQ/module_files.list.txt
Bazel/6.1.0-GCCcore-12.3.0
Java/11.0.27
ml_dtypes/.wh..wh..opq
ml_dtypes/0.3.2-gfbf-2023a
tensorboard/.wh..wh..opq
tensorboard/2.15.1-gfbf-2023a
>> Creating tarball /eessi_bot_job/eessi-2023.06-software-linux-x86_64-amd-zen2-17507550540.tar.gz from /cvmfs/software.eessi.io/versions/...
2023.06/software/linux/x86_64/amd/zen2/modules/all/Java/11.0.27.lua
2023.06/software/linux/x86_64/amd/zen2/modules/all/Bazel/6.1.0-GCCcore-12.3.0.lua
2023.06/software/linux/x86_64/amd/zen2/modules/all/ml_dtypes/0.3.2-gfbf-2023a.lua
2023.06/software/linux/x86_64/amd/zen2/modules/all/tensorboard/2.15.1-gfbf-2023a.lua
2023.06/software/linux/x86_64/amd/zen2/modules/lang/Java/11.0.27.lua
2023.06/software/linux/x86_64/amd/zen2/modules/devel/Bazel/6.1.0-GCCcore-12.3.0.lua
2023.06/software/linux/x86_64/amd/zen2/modules/tools/ml_dtypes/0.3.2-gfbf-2023a.lua
2023.06/software/linux/x86_64/amd/zen2/modules/lib/tensorboard/2.15.1-gfbf-2023a.lua
2023.06/software/linux/x86_64/amd/zen2/software/Bazel/6.1.0-GCCcore-12.3.0/
2023.06/software/linux/x86_64/amd/zen2/software/Bazel/6.1.0-GCCcore-12.3.0/bin/
2023.06/software/linux/x86_64/amd/zen2/software/Bazel/6.1.0-GCCcore-12.3.0/bin/bazel
2023.06/software/linux/x86_64/amd/zen2/software/Bazel/6.1.0-GCCcore-12.3.0/easybuild/
2023.06/software/linux/x86_64/amd/zen2/software/Bazel/6.1.0-GCCcore-12.3.0/easybuild/reprod/
2023.06/software/linux/x86_64/amd/zen2/software/Bazel/6.1.0-GCCcore-12.3.0/easybuild/reprod/easyblocks/
2023.06/software/linux/x86_64/amd/zen2/software/Bazel/6.1.0-GCCcore-12.3.0/easybuild/reprod/easyblocks/bazel.py
2023.06/software/linux/x86_64/amd/zen2/software/Bazel/6.1.0-GCCcore-12.3.0/easybuild/reprod/Bazel-6.1.0-GCCcore-12.3.0.env
2023.06/software/linux/x86_64/amd/zen2/software/Bazel/6.1.0-GCCcore-12.3.0/easybuild/reprod/Bazel-6.1.0-GCCcore-12.3.0.eb
2023.06/software/linux/x86_64/amd/zen2/software/Bazel/6.1.0-GCCcore-12.3.0/easybuild/reprod/hooks/
2023.06/software/linux/x86_64/amd/zen2/software/Bazel/6.1.0-GCCcore-12.3.0/easybuild/reprod/hooks/eb_hooks.py
2023.06/software/linux/x86_64/amd/zen2/software/Bazel/6.1.0-GCCcore-12.3.0/easybuild/easybuild-Bazel-6.1.0-20250624.084608.log.bz2
2023.06/software/linux/x86_64/amd/zen2/software/Bazel/6.1.0-GCCcore-12.3.0/easybuild/Bazel-6.1.0-GCCcore-12.3.0-easybuild-devel
2023.06/software/linux/x86_64/amd/zen2/software/Bazel/6.1.0-GCCcore-12.3.0/easybuild/Bazel-6.3.1_add-symlinks-in-runfiles.patch
2023.06/software/linux/x86_64/amd/zen2/software/Bazel/6.1.0-GCCcore-12.3.0/easybuild/Bazel-6.1.0-GCCcore-12.3.0.eb
2023.06/software/linux/x86_64/amd/zen2/software/Bazel/6.1.0-GCCcore-12.3.0/easybuild/easybuild-Bazel-6.1.0-20250624.084608_test_report.md
2023.06/software/linux/x86_64/amd/zen2/software/Java/11.0.27/
2023.06/software/linux/x86_64/amd/zen2/software/Java/11.0.27/man/
2023.06/software/linux/x86_64/amd/zen2/software/Java/11.0.27/man/ja_JP.UTF-8/
2023.06/software/linux/x86_64/amd/zen2/software/Java/11.0.27/man/ja_JP.UTF-8/man1/
2023.06/software/linux/x86_64/amd/zen2/software/Java/11.0.27/man/ja_JP.UTF-8/man1/rmid.1
2023.06/software/linux/x86_64/amd/zen2/software/Java/11.0.27/man/ja_JP.UTF-8/man1/javadoc.1

<lots of output>

2023.06/software/linux/x86_64/amd/zen2/software/tensorboard/2.15.1-gfbf-2023a/lib/python3.11/site-packages/docs/__pycache__/conf.cpython-311.pyc
/eessi_bot_job/eessi-2023.06-software-linux-x86_64-amd-zen2-17507550540.tar.gz created!
>> Cleaning up tmpdir /tmp/tmp.fecT0K6YhQ...

bedroge · 2025-06-24T12:12:29Z

Hmm, there are still some .wh.* things in the output, let me check if that's expected...

bedroge · 2025-06-24T12:24:43Z

Found it, it was due to the -o (or) in the file types, then the -name basically gets added as an and. Actually we don't need that -o, as you can list multiple types at once.

              To search for more than one type at once, you can supply the combined list of type letters separated by a comma `,' (GNU extension).

Running the updated version results in the correct output:

$ ./create_tarball.sh /tmp 2023.06 x86_64/amd/zen2 ""  /eessi_bot_job/eessi-2023.06-software-linux-x86_64-amd-zen2-17507550540.tar.gz
>> tmpdir: /tmp/tmp.bzSpAga59r
>> Collecting list of files/directories to include in tarball via /tmp/software.eessi.io/overlay-upper/versions...
handling Bazel/6.1.0-GCCcore-12.3.0
handling Java/11.0.27
handling ml_dtypes/0.3.2-gfbf-2023a
handling tensorboard/2.15.1-gfbf-2023a
wrote file list to /tmp/tmp.bzSpAga59r/files.list.txt
2023.06/software/linux/x86_64/amd/zen2/modules/all/Java/11.0.27.lua
2023.06/software/linux/x86_64/amd/zen2/modules/all/Bazel/6.1.0-GCCcore-12.3.0.lua
2023.06/software/linux/x86_64/amd/zen2/modules/all/ml_dtypes/0.3.2-gfbf-2023a.lua
2023.06/software/linux/x86_64/amd/zen2/modules/all/tensorboard/2.15.1-gfbf-2023a.lua
2023.06/software/linux/x86_64/amd/zen2/modules/lang/Java/11.0.27.lua
2023.06/software/linux/x86_64/amd/zen2/modules/devel/Bazel/6.1.0-GCCcore-12.3.0.lua
2023.06/software/linux/x86_64/amd/zen2/modules/tools/ml_dtypes/0.3.2-gfbf-2023a.lua
2023.06/software/linux/x86_64/amd/zen2/modules/lib/tensorboard/2.15.1-gfbf-2023a.lua
2023.06/software/linux/x86_64/amd/zen2/software/Bazel/6.1.0-GCCcore-12.3.0
2023.06/software/linux/x86_64/amd/zen2/software/Java/11.0.27
2023.06/software/linux/x86_64/amd/zen2/software/ml_dtypes/0.3.2-gfbf-2023a
2023.06/software/linux/x86_64/amd/zen2/software/tensorboard/2.15.1-gfbf-2023a
wrote module file list to /tmp/tmp.bzSpAga59r/module_files.list.txt
Bazel/6.1.0-GCCcore-12.3.0
Java/11.0.27
ml_dtypes/0.3.2-gfbf-2023a
tensorboard/2.15.1-gfbf-2023a
>> Creating tarball /eessi_bot_job/eessi-2023.06-software-linux-x86_64-amd-zen2-17507550540.tar.gz from /cvmfs/software.eessi.io/versions/...
2023.06/software/linux/x86_64/amd/zen2/modules/all/Java/11.0.27.lua
2023.06/software/linux/x86_64/amd/zen2/modules/all/Bazel/6.1.0-GCCcore-12.3.0.lua
2023.06/software/linux/x86_64/amd/zen2/modules/all/ml_dtypes/0.3.2-gfbf-2023a.lua
2023.06/software/linux/x86_64/amd/zen2/modules/all/tensorboard/2.15.1-gfbf-2023a.lua
2023.06/software/linux/x86_64/amd/zen2/modules/lang/Java/11.0.27.lua
2023.06/software/linux/x86_64/amd/zen2/modules/devel/Bazel/6.1.0-GCCcore-12.3.0.lua
2023.06/software/linux/x86_64/amd/zen2/modules/tools/ml_dtypes/0.3.2-gfbf-2023a.lua
2023.06/software/linux/x86_64/amd/zen2/modules/lib/tensorboard/2.15.1-gfbf-2023a.lua
2023.06/software/linux/x86_64/amd/zen2/software/Bazel/6.1.0-GCCcore-12.3.0/
2023.06/software/linux/x86_64/amd/zen2/software/Bazel/6.1.0-GCCcore-12.3.0/bin/
2023.06/software/linux/x86_64/amd/zen2/software/Bazel/6.1.0-GCCcore-12.3.0/bin/bazel

...

2023.06/software/linux/x86_64/amd/zen2/software/tensorboard/2.15.1-gfbf-2023a/lib/python3.11/site-packages/docs/__pycache__/
2023.06/software/linux/x86_64/amd/zen2/software/tensorboard/2.15.1-gfbf-2023a/lib/python3.11/site-packages/docs/__pycache__/conf.cpython-311.pyc
/eessi_bot_job/eessi-2023.06-software-linux-x86_64-amd-zen2-17507550540.tar.gz created!
>> Cleaning up tmpdir /tmp/tmp.bzSpAga59r...

bedroge · 2025-06-24T12:29:25Z

Removed one more grep ... || true for finding installation directories that are not named like .wh.*, replaced by a find with a maxdepth 0.

One more test with this version:

$ ./create_tarball.sh /tmp 2023.06 x86_64/amd/zen2 ""  /eessi_bot_job/eessi-2023.06-software-linux-x86_64-amd-zen2-17507550540.tar.gz
>> tmpdir: /tmp/tmp.ZGLM8HISv9
>> Collecting list of files/directories to include in tarball via /tmp/software.eessi.io/overlay-upper/versions...
handling Bazel/6.1.0-GCCcore-12.3.0
handling Java/11.0.27
handling ml_dtypes/0.3.2-gfbf-2023a
handling tensorboard/2.15.1-gfbf-2023a
wrote file list to /tmp/tmp.ZGLM8HISv9/files.list.txt
2023.06/software/linux/x86_64/amd/zen2/modules/all/Java/11.0.27.lua
2023.06/software/linux/x86_64/amd/zen2/modules/all/Bazel/6.1.0-GCCcore-12.3.0.lua
2023.06/software/linux/x86_64/amd/zen2/modules/all/ml_dtypes/0.3.2-gfbf-2023a.lua
2023.06/software/linux/x86_64/amd/zen2/modules/all/tensorboard/2.15.1-gfbf-2023a.lua
2023.06/software/linux/x86_64/amd/zen2/modules/lang/Java/11.0.27.lua
2023.06/software/linux/x86_64/amd/zen2/modules/devel/Bazel/6.1.0-GCCcore-12.3.0.lua
2023.06/software/linux/x86_64/amd/zen2/modules/tools/ml_dtypes/0.3.2-gfbf-2023a.lua
2023.06/software/linux/x86_64/amd/zen2/modules/lib/tensorboard/2.15.1-gfbf-2023a.lua
2023.06/software/linux/x86_64/amd/zen2/software/Bazel/6.1.0-GCCcore-12.3.0
2023.06/software/linux/x86_64/amd/zen2/software/Java/11.0.27
2023.06/software/linux/x86_64/amd/zen2/software/ml_dtypes/0.3.2-gfbf-2023a
2023.06/software/linux/x86_64/amd/zen2/software/tensorboard/2.15.1-gfbf-2023a
wrote module file list to /tmp/tmp.ZGLM8HISv9/module_files.list.txt
Bazel/6.1.0-GCCcore-12.3.0
Java/11.0.27
ml_dtypes/0.3.2-gfbf-2023a
tensorboard/2.15.1-gfbf-2023a
>> Creating tarball /eessi_bot_job/eessi-2023.06-software-linux-x86_64-amd-zen2-17507550540.tar.gz from /cvmfs/software.eessi.io/versions/...
2023.06/software/linux/x86_64/amd/zen2/modules/all/Java/11.0.27.lua
2023.06/software/linux/x86_64/amd/zen2/modules/all/Bazel/6.1.0-GCCcore-12.3.0.lua
2023.06/software/linux/x86_64/amd/zen2/modules/all/ml_dtypes/0.3.2-gfbf-2023a.lua
2023.06/software/linux/x86_64/amd/zen2/modules/all/tensorboard/2.15.1-gfbf-2023a.lua
2023.06/software/linux/x86_64/amd/zen2/modules/lang/Java/11.0.27.lua
2023.06/software/linux/x86_64/amd/zen2/modules/devel/Bazel/6.1.0-GCCcore-12.3.0.lua
2023.06/software/linux/x86_64/amd/zen2/modules/tools/ml_dtypes/0.3.2-gfbf-2023a.lua
2023.06/software/linux/x86_64/amd/zen2/modules/lib/tensorboard/2.15.1-gfbf-2023a.lua
2023.06/software/linux/x86_64/amd/zen2/software/Bazel/6.1.0-GCCcore-12.3.0/
2023.06/software/linux/x86_64/amd/zen2/software/Bazel/6.1.0-GCCcore-12.3.0/bin/
2023.06/software/linux/x86_64/amd/zen2/software/Bazel/6.1.0-GCCcore-12.3.0/bin/bazel

...

2023.06/software/linux/x86_64/amd/zen2/software/tensorboard/2.15.1-gfbf-2023a/lib/python3.11/site-packages/docs/__pycache__/
2023.06/software/linux/x86_64/amd/zen2/software/tensorboard/2.15.1-gfbf-2023a/lib/python3.11/site-packages/docs/__pycache__/conf.cpython-311.pyc
/eessi_bot_job/eessi-2023.06-software-linux-x86_64-amd-zen2-17507550540.tar.gz created!
>> Cleaning up tmpdir /tmp/tmp.ZGLM8HISv9...

trz42

Nice!

Neves-P · 2025-06-24T13:12:24Z

Here it working in the wild: EESSI/software-layer#1130 (comment)
🎉

add ' || true' to grep commands

ce7f502

trz42 approved these changes Jun 24, 2025

View reviewed changes

let find commands ignore whiteout files

baf948c

combine file types for modules to get rid of OR operator

b8c2384

get rid of one more grep command

1fa8476

trz42 approved these changes Jun 24, 2025

View reviewed changes

trz42 merged commit a676158 into EESSI:main Jun 24, 2025
45 checks passed

bedroge changed the title ~~add || true to grep commands in create_tarball.sh~~ filter whiteout files with find (instead of using grep) in create_tarball.sh Jun 25, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

filter whiteout files with `find` (instead of using `grep`) in `create_tarball.sh` #21

filter whiteout files with `find` (instead of using `grep`) in `create_tarball.sh` #21

Uh oh!

bedroge commented Jun 24, 2025 •

edited

Loading

Uh oh!

trz42 left a comment

Uh oh!

bedroge commented Jun 24, 2025

Uh oh!

bedroge commented Jun 24, 2025

Uh oh!

bedroge commented Jun 24, 2025

Uh oh!

bedroge commented Jun 24, 2025

Uh oh!

bedroge commented Jun 24, 2025

Uh oh!

trz42 left a comment

Uh oh!

Uh oh!

Neves-P commented Jun 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

filter whiteout files with find (instead of using grep) in create_tarball.sh #21

filter whiteout files with find (instead of using grep) in create_tarball.sh #21

Uh oh!

Conversation

bedroge commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trz42 left a comment

Choose a reason for hiding this comment

Uh oh!

bedroge commented Jun 24, 2025

Uh oh!

bedroge commented Jun 24, 2025

Uh oh!

bedroge commented Jun 24, 2025

Uh oh!

bedroge commented Jun 24, 2025

Uh oh!

bedroge commented Jun 24, 2025

Uh oh!

trz42 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Neves-P commented Jun 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

filter whiteout files with `find` (instead of using `grep`) in `create_tarball.sh` #21

filter whiteout files with `find` (instead of using `grep`) in `create_tarball.sh` #21

bedroge commented Jun 24, 2025 •

edited

Loading