Compact encoding for strings

[SergeyKanzhelev]

Originally from @yurishkuro

#2 (comment)

Strings are transmitted in ASCII encoding.

Is it a requirement that keys and values must be represented in ASCII? The example below could encode 34f067aa0ba902b7 as 8 bytes instead of 16 ASCII characters, but in this case an instrumentation that transfers the context from text protocol to binary protocol would need to know somehow which vendor values are actually hexadecimal representations, which doesn't sound like something we want.

{ 0,  3,  102, 111, 111,  16,  51, 52, 102, 48, 54, 55, 97, 97, 48, 98, 97, 57, 48, 50, 98, 55,
 0,  3,   98,  97, 114,  4,   48, 46, 50, 53,  }

This corresponds to 2 tracestate entries:
foo=34f067aa0ba902b7,bar=0.25

If we say that keys/values must indeed be ASCII strings, then it makes me question why we need binary encoding at all, since it does not provide any compression. It might be slightly faster to parse since the sizes are explicit.

[SergeyKanzhelev]

I was thinking about it. See w3c/trace-context-protocols-registry#4 (comment)

Charset of tracestate value

HTTP-imposed limitation on character set of the opaque value of tracestate value doesn't may not be an issue for other protocols, which generally support UTF-8 charset. This potentially may lead to various issues like one may assume UTF-8 charset for messaging protocol and will assume something like urlencoding will be applied to non-ASCII characters. However specification operates with opaque values and does not assume any encoding. So value may be simply dropped by implementation.

One of the way to solve it is in fact suggest some encoding mechanism specifically for HTTP protocol. This will, however, complicate implementations and is hard to test in existing test harness (it only supports binary now).

We can compress by utilizing space of non-printeable characters in ASCII table. It doesn't feel however we it's a lot of characters. And there is no commonly used encoding I know that will help do this. So we will need to invent a new one.

As you pointed we can simply do tracestate as a string as ASCII then. Since we do almost no compression. The only problem with this from my standpoint is the future proofing for versioning of tracestate fields which is provided by field ids. We can introduce compression later using by versioning field ID.

Also we most probably will use a similar format for correlation-context. Similar format already used in GRPC by Open Census. So it will not be completely new code to parse this field.

Do you have a strong preference one way or another? Should we look closer into compression now from your perspective?

[SergeyKanzhelev]

from @discostu105

tracestate ASCII entries: For our Dynatrace tracestate entry we would technically be able to generate a binary version too. We already use it today for tracing in various binary protocols and it has size/perf benefits. When using binary TraceContext we would be forced to use the ASCII version of our tag. Of course that's possible, but we'd lose the size/perf benefits. This is a tricky one to solve though. If a tracer needs to convert TraceContext with foreign tracestate entries from binary to text or vice-versa, it needs to know how to do that. We'd need to add a type (text or binary) to the tracestate entry.

[yurishkuro]

Maybe I am not seeing another approach, but it seems the only way to allow compact binary encoding of tracestate fields is to require that in HTTP the values of tracestate fields are base64url encoded binary blobs. This would allow standard way of decoding values into binary blobs which then can be included as-is in the binary trace-context.