Should build.zig have a programmatic interface and support the use case of acting as the low level build system of another programming language. If yes, how?

[plajjan]

Is this the right place for this issue?

I'm not sure if it's accepted to open an issue for something like this, which might be more of a question / proposal than a bug report. However, I feel this ultimately needs to be decided by core language folks, perhaps @andrewrk, thus I felt an issue was appropriate. Redirect me if necessary :)

Use of Zig build as the low level builder in Acton, another programming language

I am using the Zig build system as the low level build system in another programming language called Acton. The Acton compiler takes .act files as input, does the normal compiler things like type checking and generates a bunch of C files (one day we'll generate Zig code instead!). We then use build.zig to compile these C files, link with relevant libs etc to produce the final executable binary output. You know, the normal way of building a C program. I think this works great today with zig 0.12.0-dev.2236+32e88251e.

I'd like to gain an understanding if what I'm doing is considered kosher and supported. I am seeking to be aligned with the design of Zig build so that my use case isn't considered fringe and unsupported, on the contrary, I'd like the design of Zig's build system to be friendly to use cases like mine and support it in the long term :) I'll try to describe a few of the aspects of how Acton uses build.zig and would love feedback on whether I'm doing it the wrong way, how things will be changed in zig build and how I could better align with it. Maybe a use case like mine can also influence the design of Zig build.

An Acton project roughly looks like this:

• project/
  • deps/
    • libby (another Acton project
      • src/
        • libby.act
      • out ... looks just like in the top level project directory
  • src/
    • foo.act
    • bar.act
  • out/
    • bin/ - executable binaries go here
      • foo
    • lib/ - libActonProject.a goes here, but this is mostly not used anymore
    • types/ - intermediate output goes here, including .c, .h and .ty (which are Acton type files)
      • foo.c
      • foo.h
      • foo.ty
      • bar.c
      • bar.h
      • bar.ty

Once the Acton compiler has generated and written some C files to disk (in out/types) it invokes the builder (zig build.zig). The list of files is not specified in build.zig nor passed as arguments, instead builder walks the out/types directory and finds *.c files which it adds to the library in build.zig, see https://github.com/actonlang/acton/blob/main/builder/build.zig#L86. #18590 mentions how it might be considered poor form to be this imperative, doing IO etc. A build.zig should be declarative. I get that in the general sense but I'm not sure it helps me much writing a different program that in turn would generate a build.zig file. Quite frankly, I found it superneat that I could write my build.zig like this using the power of a real language like Zig and not be constrained to CMake or even Make. I'd really like to get if clarified if there is a future direction here. Will I/O be disallowed? Am I bound to change to generating the entire build.zig instead?

The way we invoke the zig build system is not actually with zig build. The first step of zig build is to build build.zig itself into an executable and then run that executable. Since all Acton repos are well-formed and same-formed, we actually pre-build this build.zig executable (a trick I picked up from TigerBeetle/kingprotty). This means we can always do an Acton compilation in a fraction of a second instead of waiting for build.zig to compile. An Acton project is a directory with a certain structure. It is typically used for "large" Acton projects where there are multiple modules etc. It is also possible to write single .act file programs in which case no particular directory structure is required. To simplify our builds, we actually place such single .act file programs in a temporary Acton project though so the acton compiler always sees the same Acton project directory structure. However, since we build using build.zig, we still need a build.zig with all the understanding of how an Acton C program should be compiled. Having a prebuilt builder makes this simple. In contrast, having to materialize a build.zig on disk in such a temporary project seems difficult to cache and thus leads to long compilation times upping the current sub-second compile time by at least an order of magnitude.

Acton's builder is using anonymousDependency to point out dependencies. The dependencies can be categorized into:

• external C libraries that Acton RTS depends on, like libuv for I/O, BSDNT for bigints, the Boehm GC etc
  • see e.g. https://github.com/actonlang/acton/blob/main/builder/build.zig#L261 for libuv
  • all these C libraries are shipped as source code in the Acton distribution, so you get them when you install Acton, similar to how zig ships libc sources
  • thus we never rely on zig or anything else fetching these things from the Internet for the user
  • as a side note, we maintain build.zig files for all the external C libraries we depend on, so 100% of Acton is compiled with build.zig - this unlock is a true superpower IMNSHO, like how we can do cross-compiles, out-of-tree builds, caching. good job!
• other Acton projects
  • in a projects deps/ dir, there can be other nested acton projects
  • we include and build them in our builder through a clever function that just walks the directories in deps/ and includes them as artifact to the top level project
    • since all acton projects are same-formed we can use the same build.zig as the top level project, which is what builder is based on... recursive yoo

There's a big hack here where anonymousDependency wants a relative path. Since we build a project in one place and have parts of the Acton base system in the Acton system path (e.g. /usr/lib/acton), we rewrite those absolute paths to be relative. It works but feels like a giant hack.

Then there's #18590 which is about removing anonymousDependency. The replacement seems to be a dependency() coupled with a path in build.zig.zon. This seems to require me to change so that I must generate and write build.zig and build.zig.zon files to disk for all Acton projects and their dependency Acton projects. I think that in turn will lead to cache invalidation since I'll have new build.zig files all the time. I suppose I could also implement something like anonymousDependency myself, perhaps built on the same internals of Build.zig, but it feels like I'm moving further from the supported use cases by doing that. I want to avoid having to change my implementation because Zig considers my use invalid and modifying the API. Note how I'm not asking for a stable API today, I very much understand things are still moving and I like that you guys take the time to iterate and get things right rather than try to stabilize something too early. A good design takes many iterations, but that's different to me than if I'm just not well-aligned to the design goals you have in mind.

I want to manage the cache as well. I do this today from the acton compiler, simply specifying a single cache directory for zig to use and I inspect the size of this. If it grows beyond a given threshold I wipe the whole thing and let it be rebuilt on the next acton build. I'd prefer something a little more elegant, I'm thinking LRU style. This is quite minor though and tracked in #15358

I'm sure I've missed things and there's more I should talk about and describe, but this is at least a start. I think feedback on the overall style of having a prebuilt build.zig -> builder and then doing run time discovery of things like dependencies, C files etc would be greatly appreciated. The alternative which seems more idiomatic use of Zig currently, is to write out build.zig(.zon) but I can't help but feel that it is geared towards maintainers of Zig/C/C++ libraries that manually write those files, no? Is the programmatic interface to Zig's build system meant to be generating those files? I don't want temporary files or that the user thinks "ahh, I shall edit these files to customize my build". Not having temp files avoids such confusion and possible source of bugs.