make packages fetched by zig set the executable bit based on ELF header and shebang line

[andrewrk]

Extracted from #17392.

Depends on #17462.

This issue is to eliminate this TODO comment:

zig/src/Package/Fetch.zig

Lines 1058 to 1063 in f7bc55c

	// TODO: we would like to set this to executable_bit_only, but two
	// things need to happen before that:
	// 1. the tar implementation needs to support it
	// 2. the hashing algorithm here needs to support detecting the is_executable
	// bit on Windows from the ACLs (see the isExecutable function).
	.mode_mode = .ignore,

Currently, all the mode bits of all files in a package are ignored and a default mode is used instead. This is desired, except the executable bit should be set based on whether the file is an ELF file or has a shebang line, when unpacking on a posix operating system.

[ianprime0509]

For any potential contributors, the change to respect executable status when fetching Git packages can be made here:

zig/src/Package/Fetch/git.zig

Lines 121 to 129 in 1f6d82e

	.file => {
	var file = try dir.createFile(entry.name, .{});
	defer file.close();
	try repository.odb.seekOid(entry.oid);
	var file_object = try repository.odb.readObject();
	if (file_object.type != .blob) return error.InvalidFile;
	try file.writeAll(file_object.data);
	try file.sync();
	},

The flag entry.executable is already being parsed from the tree, but currently it isn't being used in this code for compatibility with tar fetching behavior.

[travisstaloch]

i attempted this in #15382

this is the relevant code from tar.zig#L1028 in that pr. i tried linking to it but the file was too big.

if (options.mode_mode == .executable_bit_only) {
    if (std.fs.has_executable_bit) {
        // TODO - not sure using file.mode() is correct but it seems to
        //        match gnu tar behavior on linux while using
        //        header.mode does not
        const mode = try file.mode();
        var modebits = std.StaticBitSet(32){ .mask = @intCast(mode) };
        // copy the user exe bit to the group and other exe bits
        // these bit indices count from the right:
        //   u   g   o
        //   rwx rwx rwx
        //   876_543_210
        // 0b000_000_000
        const has_owner_exe_bit = modebits.isSet(6);
        modebits.setValue(3, has_owner_exe_bit);
        modebits.setValue(0, has_owner_exe_bit);
        log.debug("mode old={o} new={o}", .{ mode, modebits.mask });
        try file.chmod(modebits.mask);
    }
}

[andrewrk]

Note that std.fs.has_executable_bit is false on Windows.

[ianic]

Please help me understand why we need to read executable bit from the file system?

We have information whether the file is executable while unpacking archive (both tar and git) we can pass list of executables or list of all files with modes to the hasher. That way hasher doesn't need to walk (if it has list of all files) and extract attributes from disk. We can set executable on file systems which support that, and skip on others while having same hash on both.

[andrewrk]

Zig also needs the ability to compute the hash from a directory of files on disk.

• zig fetch [path] feature
• A future feature for verifying hash integrity of cached packages
• Generally, avoiding unnecessary design limitations

Windows has had executable bit support for decades:

tsmanner: My understanding is that the NTFS extended attributes was added for Windows to be able to mount the WSL volumes and understand the rwx bits. I should have said NTFS extended attributes, the implication is that only Windows editions that support WSL would be able to unpack things and set an executable bit.
tsmanner: In thier Windows NTFS file systems, not in the WSL vm
andrewrk: NTFS extended attributes sounds like it's worth exploring. let's have a peek at which windows version introduced that
andrewrk: https://learn.microsoft.com/en-us/windows/win32/fileio/file-access-rights-constants
andrewrk: FILE_EXECUTE For a native code file, the right to execute the file. This access right given to scripts may cause the script to be executable, depending on the script interpreter.
andrewrk: this looks old as fuck
andrewrk: yeah this API has been available since Windows XP
andrewrk: lib/std/os/windows.zig:pub const FILE_EXECUTE = 0x00000020;
andrewrk: oh look at that we already have a zig constant for it
tsmanner: Yeah, says since XP there
andrewrk: there you have it. set that bit when unpacking; read that bit when calculating the hash. that's the task

[ianic]

I tried to explain to myself how could we implement execute bit on windows.
I made small example to experiment.

NTFS has ACL (access control list) for each file. Entries in ACL can grant or deny access to the file. They are usually inherited from parent folders but can be also set on individual files. Each ACL entry (ACE) is linked to some trustee (user or group) and has access mask. Many times that access mask is set to FullControl which toggles most of the bits on in that mask. Including file_execute bit.

Here is example of reading file ACL. Initial ACL on my machine was:

tamp/foo
         inherited: true, access_mask: 1f01ff 111110000000111111111, file_execute: true, sid: /Everyone
         inherited: true, access_mask: 1f01ff 111110000000111111111, file_execute: true, sid: NT AUTHORITY/SYSTEM
         inherited: true, access_mask: 1f01ff 111110000000111111111, file_execute: true, sid: BUILTIN/Administrators
         inherited: true, access_mask: 1f01ff 111110000000111111111, file_execute: true, sid: IGORANIC5700/ianic

Access mask bits, zig constants

ACL has 4 entries each of them with full control access mask. In this case all entries are inherited from parent folder. To somehow store execute bit on file system I can add one more entry to the file ACL. Second part of the example does that:

tmp/foo with one more entry
 ==>     inherited: false, access_mask: 100020 100000000000000100000, file_execute: true, sid: /Everyone
         inherited: true, access_mask: 1f01ff 111110000000111111111, file_execute: true, sid: /Everyone
         inherited: true, access_mask: 1f01ff 111110000000111111111, file_execute: true, sid: NT AUTHORITY/SYSTEM
         inherited: true, access_mask: 1f01ff 111110000000111111111, file_execute: true, sid: BUILTIN/Administrators
         inherited: true, access_mask: 1f01ff 111110000000111111111, file_execute: true, sid: IGORANIC5700/ianic

Code is adding entry for Everyone with just file_execute bit set.

When I need to read executable bit I should read file ACL iterate over entries, find entry with some magic trustee (Everyone or current user) and access mask which has only file execute bit set and others unset. For files which has executable bit set this way that entry will probably be first in the list, for all others we need to iterate through list.

We can hack NTFS this way to store that bit but it seams to me soooo hacky, fragile and overly complicated.
Can we agree that something like this is not a way to go?