replace anytype

[the-argus]

EDIT: this issue is about both anytype and comptime T: type. All generics in zig, and how the types are constrained. for the most part, when I say anytype, just think "generics".

anytype is a tool to defer all type logic into the function, where it can be dealt with imperatively at compile time. The result is a much poorer experience with language servers and poor ordering of information: the type constraints of the arguments of a function are the first thing you want to know. And yet anytype leaves them for later.

The frustration of wanting to know what a function does and being greeted with args: anytype is what this issue is about. It sucks, and I know zig can do better. I'm going to try to summarize some previous issues on the subject, and offer some suggestions/proposals, but I don't want this issue to be closed if a suggestion is rejected, because the point of this issue is discussion, not proposal. If people like the idea then someone can make another issue marked as proposal. This issue will be closed if it's deemed not an issue but a feature, or if all solutions are out of scope.

Example

Bad example which was originally in the issue

`std.Build.dependency` is a nightmare. The function signature is `pub fn dependency(b: *Build, name: []const u8, args: anytype) *Dependency` and it takes three go-to-definitions to eventually arrive at `std.Build.applyArgs` where we can actually learn what `args` does. There are no documentation comments above any of these stdlib functions. Perhaps this could be solved by using an options struct of some kind. But nonetheless `anytype` is what got used and it's a problem that zig made it available for use in place of other options.

Ideally, we'd like some sort of annotation in the function signature of dependency which makes it clear that args should contain a target and user input options. And give a helpful error message when it doesn't, checking at the call site of dependency instead of three functions down in applyArgs.

EDIT: the above example is a bad one, as pointed out here. Consider instead this snippet from the HashMap implementation:

// If you get a compile error on this line, it means that your generic eql
// function is invalid for these parameters.
const eql = ctx.eql(key, test_key.*);

Where ctx is a generic type. This constraint is quite difficult to find if you don't find it the hard way, as it is buried in the private API of the HashMap. I think the use of anytype in the HashMap implementation is smart but it suffers in readability because there is no separation between comptime type validation logic and runtime logic (the former of which is much more important when learning the API). I think the most conservative solution to this problem would be if zig had some way of annotating a block as "for the type validation logic". Then some comptime asserts with nice error messages or comments could replace the example seen above.

Precedent

First, let's take a look at an existing solution, Zig compile-time contracts. We know type constraints can be accomplished in the language already (and that's part of why a proposed solution needs to be really good) so it's worth looking at how it would be implemented given the status quo now. A sample from zig-ctc readme:

fn signature_contract(t: anytype) contracts.RequiresAndReturns(
    contracts.is(@TypeOf(t), u8),
    void,
) {}

Compile time logic has to be inserted as a weird wrapper function around the return type. This could be helped a bit with infer T from #9260, but it still ends up pretty weird to read. This feels like something that should be a native language feature. It could be a stdlib feature instead, and maybe it should be.

In #1669, a highly related issue, a user proposed using comptime functions which return bool as types. It's readable, offers a nice experience to a user with an editor (they go-to-definition on the type, see instead it's a function, read the logic). Maybe not so nice for someone reading the code on a webpage. And, if #9260 were to be added to the language, we could remove the implicit function call and get the type of the thing in scope of the function signature:

// some hypothetical function which takes a slice of types and an input type to compare against
const oneOf = @import("std").meta.traits.oneOf;

fn doSomethingWithAStringAndANumber(str: []u8, n: oneOf(&.{u8, u16}), infer T) void {
  // do something with n. it is a u8 or a u16!
}

An improvement which I think is worth considering, in the spirit of rethinking these old, still relevant issues. But it doesn't solve the real problem, as we'll see in a moment.
Another, extremely related proposal is #6615, which proposed something like this:

fn Constraint(comptime T: type, predicate: fn (type) bool) type {
    if (!comptime predicate(T)) @compileError("wrong type");
    return T;
}

fn write2(w: Constraint(@TypeOf(w), isWriter), data: []const u8) void {
    w.write(data);
}

Additionally there were #7232 and #8008 (the latter of which was quite well thought out, proposing an @trait builtin. However. all these proposals were ultimately rejected:

There are a lot of problems with generic code. Generic code is harder to read, reason about, and optimize than code using concrete types. Even if it compiles successfully for one type, you may see errors only later when a user passes a different type. Generic code with type validation code has an even worse problem - the validation code has to match with the implementation when it changes, and there’s no way to validate that. So the position of Zig is that code using concrete types should be the primary focus and use case of the language, and we shouldn’t introduce extra complexity to make generics easier unless it provides new tools to solve these problems.

Here we come to understand that Zig's fundamental focus on clarity conflicts with generics, not that there is some problem with the proposal, really. I said the earlier example of zig-ctc felt like "should be a native language feature." I thought "hey, with some added features to Zig, this could be much more readable, and maybe also in the stdlib." But it's clear now that the zig maintainers don't want to solve that problem in the compiler to have to add a bunch of complexity to zig only to get a slightly more readable version of the exact same functionality. However, anytype is still in the language and it still is used for things other than anonymous tuples and it still sucks.

Distinct types offer a partial solution

A big part of the goal here is just to provide more readable function signatures. Yes, it would be nice to do checking in the function signature, but maybe it's enough to just provide aliases for anytype? Like a slightly better doc comment. Maybe anytype could be replaced by something like anontuple and then only allow anytype when defining type aliases. Consider #5132 (typedef for zig) and #1595 (distinct types).

Do something!

anytype is a problem when used for anything other than anonymous tuples in string formatting. Is the best answer the zig language can give doc comments? We need to come up with some functionality which actually improves the experience of writing type-constrained generic code. Something that solves problems like the type checking code buried in applyArgs.

Conclusion/Suggestion

It doesn't make sense to say "generic code is bad, its hard to reason about, so we will not add facilities to make it easier because we don't want to encourage its use" when anytype still exists and still gets used. It seems to me that people would much rather write generic code with bad facilities like anytype and deferred type-constraining logic than maintain multiple well-tested but highly duplicated implementations of the same thing. Anything short of removing comptime T: type and anytype seems unable to stop this.

Maybe the reasons we have for anytype being bad are enough to warrant reconsidering the previously closed issues like #1669? The fact that constraints on anytype are evaluated later, and not at the function where they're passed in, leads to confusing error messages. Additionally, it makes the function signature harder to read, and obfuscates information from tooling like ZLS.

My only original suggestion is that Zig could automatically generate tests for functions with anytype or comptime T: type or equivalent. These tests would only test compilation of the function with all types, not actual functionality. It would require some new builtin @constrainedTypeCompileError or something along those lines, to skip tests for types you know shouldn't work with the function. Doing such an out-of-language solution means sort of giving up and saying "there's no way to make a language with maintainable generic code, so let's just test most of the possible inputs to these bad functions." I am hoping that someone else can come up with something better.

@the-argus,

fn doSomethingWithAStringAndANumber(str: []u8, n: unsigned8BitOr16Bit) void {
    // use n. its either a u8 or a u16!
}

This syntax is certainly going to be rejected, because unsigned8BitOr16Bit looks like a concrete type, while actually being generic.

Otherwise this looks similar to #6615; have you seen that proposal?

Edit: outdated comment

[the-argus]

@zzyxyzz I have edited the original issue so now its almost totally unrecognizable. Sorry to make you read the scrapped original version, but at the time I didn't know about all the issues before me that had tried to do the same thing. In fact, in some cases, the exact same thing. GitHub's search feature is a bit of a joke if you don't know the exact name of the issue you're looking for. So thanks for linking that issue, I very well might not have found it otherwise. The new version of this issue is more of an editorial summary of the state of anytype and type-constraining features in zig, instead of a proposal like the original one was.

@the-argus,
My personal 2c on this is that Zig simply took a wrong turn with its procedural generics. In addition to what you mention (readability of type signatures, complexity of tooling, need to manually implement things that could be language features), there are also a lot of semantic dark corners where the procedural and declarative parts of the type system (and also runtime/comptime) clash. For a language that generally discourages use of generics and tries to be small enough to "fit into one's head" completely, it would have been better to have a simple parametric type system with interfaces and minimal conditional logic. (Plus introspection, but certainly not reification.)

However, at this point, the ship has sailed for any such radical redesign of the type system. FYI, the official position of the core team is that the language is mostly done and most open proposals will be rejected. As for the existing antype-style generics, a lot of people have thought long and hard about how to improve them, but without too much success. Procedural generics are kind of contagious, and it's really hard to carve out a declarative subset that is 1) general enough 2) will not cause problems elsewhere and 3) offers substantial benefits over a manual type assertion at the top of the function. My own proposal (#9260) is one of the last ones open, and it is ultimately fairly superficial.

In short, I would not get my hopes up high that there will be a substantial paradigm shift in Zig's generics. They are as powerful as they are unergonomic, but it is genuinely difficult to shift this tradeoff. Since you're not actually proposing any solution, I'd recommend closing this issue.

P.S.: Concerning the args parameter of Build.dependency, the idea is that such things will be carefully documented, even though right now they aren't.

[mlugg]

Ideally, we'd like some sort of annotation in the function signature of dependency which makes it clear that args should contain a target and user input options.

Okay, but... that's not what we want. The reason args there is anytype is because it can be literally any struct - the arguments passed to a dependency are arbitrary key-value mappings in the form of a struct value, which act the same way -Dfoo=bar options do to the "root" build function. By convention, we commonly pass target and optimize to dependencies to be picked up in the dependency by standardOptimizeOption and standardTargetOptions, but this is not a requirement, and there are cases where you wouldn't want this. The args parameter there is just like the args parameter in std.fmt.format.

[...] and the zig maintainers don't want to solve that problem in the compiler.

To be clear, none of this is really a problem of implementational complexity - any of the ideas talked about here could be integrated into the compiler fairly trivially. The problem is not that we don't want to bother solving the problem in the compiler, but instead that we don't necessarily think any such idea is a good fit for Zig as a language.

It doesn't make sense to say "generic code is bad, its hard to reason about, so we will not add facilities to make it easier because we don't want to encourage its use" when anytype still exists and still gets used.

I don't think this is really anyone's position on the matter, at least not to this extreme. It's not controversial to say that generic code can be useful, and often clearly the right tool for the job. The danger is that if Zig has language-level support for more complicated generic patterns - the kind of thing that requires a lot of complex validation logic - then we implicitly encourage such constructs, which often results in a web of unnecessary abstraction - look at the modern C++ standard library for instance. Status quo Zig allows these patterns, and allows you to easily integrate validation logic when you need (see the std.HashMap "adapted" APIs for a great example of this), but does not implicitly encourage such patterns, so they are only used when they are clearly the right tool for the job.

anytype is a problem when used for anything other than anonymous tuples in string formatting. Is the best answer the zig language can give doc comments?

I'd just like to close out by saying that as someone who's been using Zig as their primary language for a few years now, I've never actually had any major issues with anytype. It's quite rare to see generic functions which use it (much more common are comptime arguments, particularly comptime T: type), and when I do see them, it's not the kind of thing which could be trivially specified in a type signature: you'd need a function containing separate validation logic anyway (e.g. the adapter API), so being able to move this validation up one line into the signature rather than the body doesn't seem in any way helpful. The other case where I commonly see anytype is in functions polymorphic over numeric types (e.g. a lot of functions in std.math), but here it's always obvious from context what is valid: this being explicit in the signature would not really be helpful, because there's no confusion to begin with.

(As a note, I do remain in support of the infer T proposal, but that's essentially a small syntactic sugar to avoid using @TypeOf all over the place - I don't like the extensions people have suggested to that proposal for things like reversing type-creating functions.)

[the-argus]

Ideally, we'd like some sort of annotation in the function signature of dependency which makes it clear that args should contain a target and user input options.

Okay, but... that's not what we want. The reason args there is anytype is because it can be literally any struct - the arguments passed to a dependency are arbitrary key-value mappings

I think it is odd that args accepts both target and the build options all in one big tuple. I can see that they are all quite literally command line arguments, though, so args is descriptive. I guess this example is just flawed and more of just a problem with lack of documentation.

if Zig has language-level support for more complicated generic patterns - the kind of thing that requires a lot of complex validation logic - then we implicitly encourage such constructs, which often results in a web of unnecessary abstraction - look at the modern C++ standard library for instance. Status quo Zig allows these patterns, and allows you to easily integrate validation logic when you need (see the std.HashMap "adapted" APIs for a great example of this), but does not implicitly encourage such patterns, so they are only used when they are clearly the right tool for the job.

From std.HashMap:

// If you get a compile error on this line, it means that your generic eql
// function is invalid for these parameters.
const eql = ctx.eql(key, test_key.*);

Is this really... good? bits of the type validation end up buried in the code.

What if you could just go-to-definition on the anytype or type argument and hop right to the code that constrains the type? That's purely a QOL improvement just by allowing some way of annotating type-constraining code. I think that if something like interfaces were to be added, it would encourage people to constantly look for commonalities and sometimes over-architect their code. But would adding a way of defining where type-constraining code is found do the same?

It's quite rare to see generic functions which use it (much more common are comptime arguments, particularly comptime T: type),

I think that anytype and comptime T: type are effectively interchangeable when it comes to most of the problems I posed.

it's not the kind of thing which could be trivially specified in a type signature

This is true! And to allow for specifying such complex constraints, we should not create a complex type declaration syntax. And then make it accidentally turing complete. And then make that a feature. That would be bad. Instead, we could just... put the type constraining code in a block and give it a helpful name.

[the-argus]

However, at this point, the ship has sailed for any such radical redesign of the type system. FYI, the official position of the core team is that the language is mostly done and most open proposals will be rejected.

This issue was sort of a cry for help... I know proposals are not being accepted but to me generics feel like a huge hole in what is otherwise an awesome language. I just want perfection, is that too much to ask? /s

In short, I would not get my hopes up high that there will be a substantial paradigm shift in Zig's generics. They are as powerful as they are unergonomic,

Reminds me of C++.

Since you're not actually proposing any solution, I'd recommend closing this issue.

Yeah... all I've got is the suggestion to test all type combinations. I'm going to leave it open for a day or so but then I will close it based on your input.

P.S.: Concerning the args parameter of Build.dependency, the idea is that such things will be carefully documented, even though right now they aren't.

yay :)

[iacore]

You can avoid anytype if you put everything inside of fn xxxx(comptime T: type) { ... }.

Then, you have to use xxxx(WHATEVER_HERE).f(foo) instead of f(foo).

[andrewrk]

proposals are not being accepted

I see you have some actual experience using zig in earnest so it's ok 👍

[iacore]

proposals are not being accepted

I see you have some actual experience using zig in earnest so it's ok 👍

@andrewrk Can you update the issue template to clarify this? I was banned once so it seemed arbitrary.

[marler8997]

Thanks for providing a nice history/summary!

Let me restate the fundamental problem to set the stage for an idea:

anytype makes it hard to figure out what a parameter type is supposed to be.

I would categorize the following as symptoms of this fundamental problem:

• it's hard for developers to know what they can pass into a function
• it's hard for IDE's doc generators to know what to do with that parameter
• it's hard for the compiler to provide good error messages, is it a problem with the parameter type or in the function itself?

So far the ideas we've seen give the developer additional tools they can use to help alleviate the fundamental problem. However, I wonder if we could improve the situation without requiring any new langauge syntax nor any extra specification from library authors? It seems to me there's already alot of information about an anytype parameter that can already be derived. Take this example:

fn example(c: anytype) void {
    c.foo();
}

It's clear that c is some type that needs to have a function foo. My understanding is that Zig re-analyzes generic functions for every unique set of concrete types it receives, however, one could imagine an additional pass that attempts to analyze the nature of it's generic arguments, and derives a contract for them. This could be used to make error messages much nicer, i.e.

// call the example above
example(42);
// zig: error: function `example` requires a type that has a function 'foo' but you provided comptime_int

This could also be used in doc generation:

fn example(c: anytype)

c: a struct that has a method `fn foo() void`

Now, this is a very simple example. Deriving a contract for a type based on code is hard to generalize I think. Take this example:

fn example(c: anytype) void {
    if (some_condition) {
        return c.foo();
    } else {
        return c.bar();
    }
}

Now what is the compiler to do? c might need to have a method foo or a method bar. Now make some_condition some arbitrarily convoluted comptime code and it's unclear what should be done.

I think this is a bit of an unexplored space, but what's nice is if we can find improvements here, you make the lives of all Zig developers better at no cost to them, no new language syntax to learn/use. Also keep in mind that if we don't explore this direction, then any new feature that we add to the language to alleviate the problem may actually be rendered unnecessary because the code was already giving the compiler all the information it needed.

@marler8997,
Such analysis would be limited to simple cases, because nested functions, comptime conditionals and custom compile errors make it infeasible to derive type constraints from code in general.

This is a problem shared by all languages that do not have formal interfaces and use some form of duck typing for polymorphism (Python, Julia, C++ templates, etc.): faults can only be consistently detected for concrete instantiations, so you often have an error bubbling up from somewhere deep in the call stack and have no help in diagnosing the actual problem. Zig is actually not the worst offender in this regard, because building very many layered comptime abstractions is not idiomatic, but the basic problem is the same.

The only real countermeasure is to provide detailed (and up-to-date) documentation and manually insert comptime checks.

[marler8997]

I see the concerns but I have a feeling we could do a lot more here than it may seem at first. For example, nested functions doesn't seem like an issue, the caller simply inherits the contract established in the callee. It's hard to say where the real pain points will be so let's explore. Let's take a look at std.mem.len:

pub fn len(value: anytype) usize {
    switch (@typeInfo(@TypeOf(value))) {
        .Pointer => |info| switch (info.size) {
            .Many => {
                const sentinel_ptr = info.sentinel orelse
                    @compileError("invalid type given to std.mem.len: " ++ @typeName(@TypeOf(value)));
                const sentinel = @ptrCast(*align(1) const info.child, sentinel_ptr).*;
                return indexOfSentinel(info.child, sentinel, value);
            },
            .C => {
                assert(value != null);
                return indexOfSentinel(info.child, 0, value);
            },
            else => @compileError("invalid type given to std.mem.len: " ++ @typeName(@TypeOf(value))),
        },
        else => @compileError("invalid type given to std.mem.len: " ++ @typeName(@TypeOf(value))),
    }
}

The way I imagine this working is that we analyze the len function like we would with any concrete type except use special Contract objects for every generic parameter. The analyzer sends these Contract objects all the operations the code performs on them, and it tracks those operations.

We start our "Generic Type Analysis" pass of the len function and come to this expression:

switch (@typeInfo(@TypeOf(value)))

Since value is a Contract object, the @TypeOf builtin will return a special object, let's call it TypeContract that will forward any operations performed on it to the underlying Contract object, except that those operations put constraints on the type of value rather than on value itself.

The same is done for @typeInfo, where operations performed on it will apply to a TypeInfoContract object and put constraints on the type info of the type of value rather than value itself.

So next the analyzer sees that we call switch on the TypeInfoContract for value. Zig can immediately see that the only valid tag is Pointer, because all other tags result in a @compileError. So the analyzer tells the TypeInfoContract that it must be a pointer, which tells the TypeContract object it must be a pointer, which tells Contract it must have a pointer type! So now if we see this code, zig has everything it needs to generate its own error message!

std.mem.len(0);
// error: std.mem.len requires a pointer but received comptime_int

Keep in mind that with this sort of analysis, len no longer needs to provide its own error message in this switch statement, the compiler can already derive the proper error message.

pub fn len(value: anytype) usize {
    switch (@typeInfo(@TypeOf(value))) {
        .Pointer => {
            // ...
        },
        else => @compileError(), // no need for message!
    }
}

I don't know about you but this alone already seems promising enough to explore. Even if it's not possible to solve this in every case, it could make using the language greatly improved I think.

[marler8997]

Some afterthoughts. Note that since this idea adds an "analysis pass" on every generic function, we can not only detect issues with the caller types, we can also detect errors in the function body regardless of the parameter types passed in.

fn example(c: anytype) void {
    c.foo(x);
}

In this example, when zig does its "analysis pass", it sees that x is undefined regardless of what c is. Zig can now emit an error with the example function without it even being instantiated and knows it doesn't need to involve any of the caller's context in it's "notes"..the buck stops at this function.

Also to add to my previous comment. Capturing type constraints like I outlined above would explode if you try to perfectly capture all the specific cases code operations can affect a type, but, keep in mind that this analysis doesn't have to be perfect. The type constraints don't affect compilation, they only affect error messages and documentation. So maybe you have some sort of nested branching code that's 10 levels deep based on the type of a value, so you end up with thousands of possible type combinations depending upon the input, but we don't need to worry about perfectly tracking all those combinations. Instead we can reduce them to a flat set of valid/invalid general types. For example, maybe the type of x needs to be a u32 if y is greater than 0 but then x should be a []const u8 if y is less than -10 but only on sundays....we don't really need to capture all that detail. Our analysis captured that x needs to be a u32 or []const u8 and that's probably all we need.

Here's a shorter way of saying the same thing:

We don't need to know exactly WHEN an anytype can or can't be something...we only need to know IF it can. I think not having to know the WHEN might be the key to preventing this from exploding.