Builds are no longer reproducible

[leighmcculloch]

Zig Version

0.10.0

Steps to Reproduce and Observed Behavior

const std = @import("std");

pub fn main() void {
    std.debug.print("{any}", .{1});
}

Building that code file multiple times with release results in different binaries every time:

❯ rm main ; zig build-exe -O ReleaseSafe main.zig && mv main main1 && shasum -a 256 main1
89d9955ed3278f2e3094d895bf6edc35ea99fdcf9a451a79566fdb9d9c38ef6b  main1
❯ rm main ; zig build-exe -O ReleaseSafe main.zig && mv main main2 && shasum -a 256 main2
0ce290b125384145b8ef40471c74a6e2befe2bb29bc1a26db93fb55ca6dd4a91  main2

Upon a diff inspection, the difference is minimal and consistently In these same locations:

git diff --no-index =(hexdump -C main1) =(hexdump -C main2)

diff --git a/tmp/zshvBoeFg b/tmp/zshy7Dwt7
index 9f7964d..8ba3816 100644
--- a/tmp/zshvBoeFg
+++ b/tmp/zshy7Dwt7
@@ -101,7 +101,7 @@
 00000640  00 00 00 00 00 00 00 00  32 00 00 00 20 00 00 00  |........2... ...|
 00000650  01 00 00 00 00 00 0d 00  00 00 0d 00 01 00 00 00  |................|
 00000660  03 00 00 00 00 00 00 00  1b 00 00 00 18 00 00 00  |................|
-00000670  ef ee 6b d6 b6 c7 2c 65  41 c1 12 47 6d f0 c9 ff  |..k...,eA..Gm...|
+00000670  b2 56 49 d3 1d c2 36 c5  1a dc d6 a6 1a 91 41 12  |.VI...6.......A.|
 00000680  0c 00 00 00 38 00 00 00  18 00 00 00 02 00 00 00  |....8...........|
 00000690  00 00 27 05 00 00 01 00  2f 75 73 72 2f 6c 69 62  |..'...../usr/lib|
 000006a0  2f 6c 69 62 53 79 73 74  65 6d 2e 42 2e 64 79 6c  |/libSystem.B.dyl|
@@ -9556,7 +9556,7 @@
 00027d60  0d 19 00 00 0e 0c 00 00  0c 61 02 00 01 00 00 00  |.........a......|
 00027d70  1e 19 00 00 64 00 00 00  00 00 00 00 00 00 00 00  |....d...........|
 00027d80  20 19 00 00 64 00 00 00  00 00 00 00 00 00 00 00  | ...d...........|
-00027d90  25 19 00 00 66 00 01 00  50 20 98 63 00 00 00 00  |%...f...P .c....|
+00027d90  25 19 00 00 66 00 01 00  5e 20 98 63 00 00 00 00  |%...f...^ .c....|
 00027da0  00 00 00 00 2e 01 00 00  c0 06 00 00 01 00 00 00  |................|
 00027db0  26 00 00 00 24 01 00 00  c0 06 00 00 01 00 00 00  |&...$...........|
 00027dc0  00 00 00 00 24 00 00 00  30 00 00 00 00 00 00 00  |....$...0.......|

This build is on macOS arm64.

Expected Behavior

According to https://ziglang.org/documentation/master/#Build-Mode the ReleaseSafe and other release builds are Reproducible builds.

[leighmcculloch]

Running the same commands on a homebrew/brew Docker arm64 image, with the same version of zig installed via brew install zig, results in consistent builds:

linuxbrew@942f29014363:~$ rm main ; zig build-exe -O ReleaseSafe main.zig && mv main main1 && shasum -a 256 main1
1cdab95380a8f4f9503757d64379e9f3fe6d30566e2d5eaa6fa366a039c1da17  main1

linuxbrew@942f29014363:~$ rm main ; zig build-exe -O ReleaseSafe main.zig && mv main main2 && shasum -a 256 main2
1cdab95380a8f4f9503757d64379e9f3fe6d30566e2d5eaa6fa366a039c1da17  main2

[matu3ba]

Can you bisect this?

[andrewrk]

Before closing this issue, this diff should be applied so that we gain CI coverage and catch regressions:

diff --git a/ci/aarch64-macos.sh b/ci/aarch64-macos.sh
index 337c2bfa8..2fe38f21a 100755
--- a/ci/aarch64-macos.sh
+++ b/ci/aarch64-macos.sh
@@ -52,3 +52,19 @@ stage3-release/bin/zig build test docs \
 
 # Produce the experimental std lib documentation.
 stage3-release/bin/zig test ../lib/std/std.zig -femit-docs -fno-emit-bin --zig-lib-dir ../lib
+
+# Ensure that stage3 and stage4 are byte-for-byte identical.
+stage3-release/bin/zig build \
+  --prefix stage4-release \
+  -Denable-llvm \
+  -Dno-lib \
+  -Drelease \
+  -Dstrip \
+  -Dtarget=$TARGET \
+  -Duse-zig-libcxx \
+  -Dversion-string="$(stage3-release/bin/zig version)"
+
+# diff returns an error code if the files differ.
+echo "If the following command fails, it means nondeterminism has been"
+echo "introduced, making stage3 and stage4 no longer byte-for-byte identical."
+diff stage3-release/bin/zig stage4-release/bin/zig
diff --git a/ci/x86_64-macos.sh b/ci/x86_64-macos.sh
index f09121ccd..9d9cdb9d6 100755
--- a/ci/x86_64-macos.sh
+++ b/ci/x86_64-macos.sh
@@ -60,3 +60,19 @@ stage3-release/bin/zig build test docs \
 
 # Produce the experimental std lib documentation.
 stage3-release/bin/zig test ../lib/std/std.zig -femit-docs -fno-emit-bin --zig-lib-dir ../lib
+
+# Ensure that stage3 and stage4 are byte-for-byte identical.
+stage3-release/bin/zig build \
+  --prefix stage4-release \
+  -Denable-llvm \
+  -Dno-lib \
+  -Drelease \
+  -Dstrip \
+  -Dtarget=$TARGET \
+  -Duse-zig-libcxx \
+  -Dversion-string="$(stage3-release/bin/zig version)"
+
+# diff returns an error code if the files differ.
+echo "If the following command fails, it means nondeterminism has been"
+echo "introduced, making stage3 and stage4 no longer byte-for-byte identical."
+diff stage3-release/bin/zig stage4-release/bin/zig

[kubkon]

Just an FYI that #13964 should fix this. This is what I'm getting locally on x86_64-macos with changes in the PR:

❯ rm main ; ../../zig/build/stage4/bin/zig build-exe -O ReleaseSafe -fstrip main.zig && mv main main1 && shasum -a 256 main1
rm: main: No such file or directory
4dee3fcb60ad05e894ccc7b549aa232497f8eac3469c549ca77ddb0f9b128c5f  main1

❯ rm main ; ../../zig/build/stage4/bin/zig build-exe -O ReleaseSafe -fstrip main.zig && mv main main2 && shasum -a 256 main2
rm: main: No such file or directory
4dee3fcb60ad05e894ccc7b549aa232497f8eac3469c549ca77ddb0f9b128c5f  main2

Please do note that we need to opt-in into debug-info strip as otherwise we end up with filesystem non-determinism in the binary which is to be expected. According to Apple, you should use LC_UUID value if you want to compare binary with and with debug info stripped. We now calculate the UUID value excluding sections in the binary which contribute to non-deterministic results.

I will also post results for aarch64-macos momentarily.

[kubkon]

aarch64-macos results:

❯ rm main ; ../../zig/build/stage4/bin/zig build-exe -O ReleaseSafe -fstrip main.zig && mv main main1 && shasum -a 256 main1
rm: main: No such file or directory
6f8b637c0d53e67c28509b8daa86528c9536bd1e6f600fe6c91bdeb653656be7  main1
 
❯ rm main ; ../../zig/build/stage4/bin/zig build-exe -O ReleaseSafe -fstrip main.zig && mv main main2 && shasum -a 256 main2
rm: main: No such file or directory
6f8b637c0d53e67c28509b8daa86528c9536bd1e6f600fe6c91bdeb653656be7  main2

[kubkon]

As #13964 has now been merged, I will close this issue as resolved. Feel free to reopen or submit a new issue if you feel it hasn't though.