Tags

[BraedonWooding]

This was discussed originally here, and elsewhere :).

Motivation

I've been looking at implementing lambda functions (by first making all functions anonymous) but I feel like I should begin with something a little easier as to help me understand the codebase; I've recently had need for reflection metadata in a codegen which generates getters/setters (amongst other things) for structs in relation to a stack based system for a markup language that translates to a IR/bytecode representation, anyways it would be nice to allow users to disable/hide this generation with a hide tag (as well as customise such as link getters/setters to other functions). Amongst other tags, regardless this would be a nice feature I'm sure many authors would enjoy. I'll be looking towards implementing this, this upcoming weekend but I wanted to gather opinion on syntax :).

Proposal

The original proposal was this;

const Foo = struct {
    x: u32 @("anything") @(bar()),
};

fn bar() i32 {
    return 1234;
}

However this proposes a problem of how one would access the tags, since they aren't purely the same type; in the majority of cases (specifically I can't think of a case where this isn't true) you just want a string tag not a 'boolean' tag since in reality having something like @(true) is meaningless compared to @("Hide") so I propose the following restriction; all tags are just strings and @("X") is replaced with @tag("X") this makes it a little simpler to read and a lot simpler to actually handle.

const Foo = struct {
  x: u32 @tag("A") @tag("B"),
};

Also tags will be able to be applied to almost anything that is globally scoped that is functions (global), structs (note: variables can't have tags) having something like;

fn X() void @tag("MyFunc") {
    ...
}

You would access it like;

// For Struct
for (@typeInfo(Foo).Struct.fields) |field| {
    for (field.Tags) |tag| {
       // Do whatever with tag
    }
}

// For function
for (@typeInfo(X).Fn) |tag| {
    // Do whatever with tag
}

You could also query all objects that have a certain tag like;

comptime const query = @withTag("A"); // returning typeInfo

Now this would look through each typeInfo available returning a static array of all that have it, however we could also maintain a map of each tag to a list of typeinfo making this much much more efficient but increase the size of the running compiler (I feel a valid tradeoff).

Actual Changes

• Syntax Change: allow @tag(...) after types in global 'objects' (functions, structs), currently only @tag(...) but probably add syntactically support for any attribute (in terms of format @<...>(<...>) but lexically only support tag, this would allow us to expand it later for other attributes if wanted
• Builtin.zig change (autogen that is); add 'tags' field to StructField, and Fn that is a string slice (u8 const)
• Add @tag(tagName: []const u8) and @withTag(tagName: []const u8)->[]const @import("builtin").TypeInfo
  • Note: @tag is not grouped under attributes perhaps but rather is separate as it doesn't have a return type

[bheads]

I was just thinking that I could use attributes the other day, there are uses for more complicated tags. Ex: you could tag members of a struct for serialization: name, type, optional, required flags ect..

I'm not sure I understand the motivation for tags. Could you give a written, practical example maybe?

after all I see this sounds easy for you and you want to continue working on harder topics which is great 😃

thanks

[PavelVozenilek]

I do not see where this would be more useful than confusing. Java has (or had) something similar, and it was big mess to use. Project guidelines/unwritten rules pushed people to place annotations everywhere, no matter how silly it was. Most even didn't know why they were doing it.

Invisibly generated things are evil. For example, serialization is really better to be written down manually, instead of relying on black magic.

If there's area where invisibly generated trickery may be of some use, it would be "aspects", and even this mostly to add debugging checks. Aspects, fortunately, do not require tagging things.

---

However, there's one specific case where tags should be allowed, and available for compile time and runtime inspection: in tests. My proposals: #567 and #1010. Also the "this test must fail to compile" feature (#513) could be supported in the same way.

[BraedonWooding]

@bheads what's a use-case for a required flag, couldn't you just state that if the flag isn't given it is false by default and if it is given it is true? Things like names and so on can be implemented maybe even like an arg parser like @tag("SetName:Bar"), the important thing is to keep it simple because else it gets really difficult to use because the types don't match and because annoying to go through and iterate tags.

@monouser7dig a good example would be something like;

const OptionsSpec = struct { 
  h: ?bool @tag("flag"),
  fileName: []u8 const,
  inputFiles: [][]u8 const @tag("many"),
  state: i32 @tag("Only: 1, 0, -1"),
};

Another good use case would be having just a simple @tag("Hide") flag to hide a certain field from serialisation.

[BraedonWooding]

@PavelVozenilek I definitely understand your plight with Java's annotations, but keep in mind this ISN'T annotations, this is merely extra meta data. For example in java you have stuff like @throws, @doesThisWeirdThing and so on to describe how something behaves, in this tag system it would ONLY be used for allowing the user to do comptime reflection to perform certain actions such as setup serialisation scripts (as above), and perhaps you want to put all functions that have the @tag("Console") into a dictionary for runtime access so you would have something like;

fn ClearScreen() void @tag("Console") { .. }

This way you would be able to add all the entire the dictionary with a simple for loop using the query function INSTEAD of having;

loadConsoleCommand(Clear);
loadConsoleCommand(Other);
loadConsoleCommand(A);
loadConsoleCommand(B);
loadConsoleCommand(C);

Not having 200 lines of 'loads' and just having a simple for loop is a HUGE thing, I've done this so many times in games that it really is an important use-case :).

So while I understand your problem with Java, I kind of really want to enforce that the intention is not at all aligned with java and much more with C# attributes (have a look at them if you have time) which are ONLY ever used when the user wants to have a unique attribute for stuff like the function thing I gave above, OR when doing certain stuff like serialization.

So I don't see people doing ANY stuff like the Java annotation system, as the only use-case in the std library I can see right now would be for JSON serialization of structs.

okay so a tag is essentially just a string, I'd agree that this is actually easy to read.

I see the point for serialization or the game example..although in the game example I think a different overall data structure may be the real answer but I've not dealt with the same situation myself so could be off.

The @tag("Only: 1, 0, -1") should really be an enum or in other cases a custom type (-> enforced by compiler) though and this @tag("many") does not really contain any meaning?.

what confused me in the beginning was this part @(bar()

const Foo = struct {
   x: u32 @("anything") @(bar()),
                         ^~~~
};

what was this supposed to do in the old proposal?

[BraedonWooding]

@monouser7dig we needed a string to function dictionary because we wanted to allow users to type commands and have them run, maybe there was a better data structure regardless there was a need to have a tag based system.

The bar was to do with the old proposal (under the Clap library issues).

Keep in mind that both of those are context sensitive, they make sense in relation to the library for example the clap library allows you to specify that something is a 'many' argument i.e. is like -s a -s b -s c and this was just chosen to simulate that; a better name could be chosen but I was just trying to keep to that. The case of Only actually maybe indicates the sufferings of using a single type ([]u8) to represent all situations as yes a better case would be enum but how they can be implemented efficiently on the builtin side is the difficult position, unless we want something like;

struct Tag {
    ptr: usize,
    T: type,
}

Which would allow you to ptr cast it back into the original value, however this has problems of allocation and requiring unsafe ptr semantics.

If you can think of a way to have multiple types in such a way that is safe, the only other way I can think of it is have Tag look something like;

struct Tag {
    name: []const u8,
    data: []const u8,
}

You could use the data field to do things like store a float, or an enum or anything and thus the syntax would be; @tag("only", i32(-1), i32(0), i32(1)), or something like that, the data field could even represent anything really a float or whatever; we would take in the arguments as var args storing each argument in the data array, which would be 4*3 (12) bytes; this would be annoying to handle however and is very prone to errors and I don't particularly like this.

I think perhaps the best way to approach this is just encourage string tags that are like hide or many and discourage ones like only: -1, 0, 1. Then later on (or now) if someone has a better idea to allow for the extra types then that can be implemented.

[tgschultz]

I've spent some time thinking about this, and I think it might not even be necessary to introduce a new builtin for most cases. Consider that Zig has a zero-sized void type.

const Foo = struct {
    x: u32,    _note_anything: void, _func_bar: void,
    
};

fn bar() i32 {
    return 1234;
}

Now Foo will have these names in @typeInfo(Foo).Struct.Fields and we can identify them by their void type and leading _. They can be associated to the x field by their ordering. One issue with this method is that you can't have two fields named identically, so this kind of thing has to happen:

const Pos = struct {
    x: f32, _x_pack_fp2dot14: void, //or _pack_fp2dot14_0
    y: f32, _y_pack_fp2dot14: void,  //or _pack_fp2dot14_1
};

This could potentially be remedied by allowing field names of void type to be repeated, but I'm not sure what consequences might result elsewhere from that. This also couldn't be used to annotate things that aren't fields (although it does make multiple tags simple).

So from there, my thinking is that what we actually want to annotate is not the field itself, but the type of the field. Currently type aliases have no meaning, they are just different names for the same thing. If we could, at comptime, determine what alias was used for a type then we can just operate on that.

const fp2dot14 = f32;

const Pos = struct {
    .x: fp2dot14,
    .y: fp2dot14,
};

Now in comptime code I could look at the alias with something like @aliasNameOf(T) and know how to serialize the field for use in the netcode. Still no way to apply it to a function though, since we can declare a type alias for a function type, but we can't define a function using it.

My personal preference is not to add a builtin if it can be reasonably avoided, but if we did do that, I propose a small change to the syntax presented:

const Pos = struct {
    .x: @tag(f32, "fp2dot14"),
    .y: @tag(f32, "fp2dot14"),
};

//or we can make a type alias of the tagged type:
const u32be = @tag(u32, builtin.Endian.Big);

//and have multiple tags
const u12be_lsb = @tag(u12, builtin.Endian.Big, BitPacking.LSBFirst);

//functions don't quite work the same, but:
fn add(x: i32, y: i32) i32 {
    return x + y;
}
@fnTag(add, Console.TwoArgs);

[Hejsil]

It was discussed in #676, that we could allow _ as a name in structs which could be used for padding and such. That, together with zero sized types, could be used for everything tags would do:

const S = struct {
    _: json.Ignore,
    ignore_me: u8,

    _: json.CustomFormatter(formatEnum),
    some_enum_field: E,
};

We can ever expand this to all definitions if we allow _ in global scope.

const _ = ConsoleCommand{};
fn someConsoleCommand(c: *Console) void {
    c.print("Hello World\n");
}

With #1047, the @withTag could even be implemented in userland, though it would only operate on type instead of the whole project (which I don't think is realistically possible anyway).

// meta.zig
pub fn withTag(comptime Namespace: type, comptime TagType: type) []const TypeInfo.Definition {
    ...
}

// commands.zig
const commands = comptime meta.withTag(this, ConsoleCommand);

[tgschultz]

I knew I just hadn't spent enough time thinking about it. It didn't occur to me that empty structs are also size zero.

[bheads]

I was thinking something like this:

something like json_required would be a parsing error if not found, or json_option is okay is missing
json_id would change the mapping, json_type changes parsing types, ect...

const RpcRequest = struct {
    id: u64 @tag("json_required") @tag("json_id", "request id") @tag("json_type", "string"),
    method_name: []u8,
    args: [][]u8, @tag("json_optional")
};

...
const rpc_request = try json.parse(RpcRequest, some_json_string);

[thejoshwolfe]

I don't like the idea of using empty fields to convey something that isn't related to fields. That doesn't convey intent very well.

I was recently confused by C++ code that used std:: piecewise_construct as a parameter to a function, even though it doesn't pass any data to the function. A parameter should pass data, and a field should hold data. That's the intent of those constructs.

@BraedonWooding

The bar was to do with the old proposal (under the Clap library issues).

is this something that would be impossible now? I wanted to ask because maybe the current way would miss that (important?) use case.