From 8587e0708b0d78196332359753254dd490ac41ee Mon Sep 17 00:00:00 2001 From: Sergey Kanzhelev Date: Thu, 13 Dec 2018 10:48:35 -0800 Subject: [PATCH 1/6] amqp notes --- extension-amqp.html | 12 +++--------- spec/AMQP/20-AMQP_FORMAT.md | 32 +++++++++++++++++++++++++++++++- 2 files changed, 34 insertions(+), 10 deletions(-) diff --git a/extension-amqp.html b/extension-amqp.html index 8cc1aa4b..221cdfe8 100644 --- a/extension-amqp.html +++ b/extension-amqp.html @@ -16,16 +16,10 @@ w3cid: "103966" }, { - name: "Morgan McLean", - company: "Google", - companyURL: "https://google.com", + name: "Clemens Vasters", + company: "Microsoft", + companyURL: "https://microsoft.com", w3cid: "104128" - }, - { - name: "Alois Reitbauer", - company: "Dynatrace", - companyURL: "https://dynatrace.com", - w3cid: "48276" }], github: { repoURL: "https://github.com/w3c/trace-context/", diff --git a/spec/AMQP/20-AMQP_FORMAT.md b/spec/AMQP/20-AMQP_FORMAT.md index b503472f..5c43ede2 100644 --- a/spec/AMQP/20-AMQP_FORMAT.md +++ b/spec/AMQP/20-AMQP_FORMAT.md @@ -1,3 +1,33 @@ # AMQP format -TBD \ No newline at end of file +The AMQP binding will define for traceparent and tracestate to be added to the +message in the application-properties section (as “w3c:tracestate” and +“w3c:traceparent”) if they are added by the original message publisher. Once the +message has been created, it’s no longer permissible to edit the bare message, +so if it were necessary to annotate the message inside the middleware as it +transits, that will happen in the “message-annotations” section, using the +“application-properties” as a base. The ultimate receiver can then pick between +the full context including infrastructure and the pure app context by either +looking into “message-annotations” or “application-properties”. + +1. *Protocol justification*. I suggest just reference W3C specs as they did in cloud events. And keep names of “headers”. +2. *Size-sensitive environments*. There is a good reason to keep the format and sizes unchanged. For the size-sensitive environments +If you ONLY INITIATE the message outside of existing context – there may be a possibility to save on size by truncating fields. +Discuss how to use binary protocol instead of textual. +Differences with message-id. Discuss how message-id is a deduplication concept where Trace-Context identifies the context of worker who either put or read the message. +Batching of messages – recommended semantics of TraceContext fields when batching is happening. + + +Failed post or get – what is relationship between failed operation trace + context and message trace context. +Log entire app properties to get correlation. + +Mutation of trace context by queue provider – should we allow/recommend to mutate trace context fields by trace provider. Especially for the span-id. + Security/privacy consideration – we can refer to W3C spec again. And adjust + it if needed. + +application properties is "unmodifiable" so "message-annotations" overrides +those in application + +traceparent 1.6.22 (list of binary) or binary - make rationale +tracestate 1.6.23 (map of string/string type) \ No newline at end of file From f70357c24ff1114431e52db2861533c7d1217319 Mon Sep 17 00:00:00 2001 From: Sergey Kanzhelev Date: Fri, 14 Dec 2018 09:15:53 -0800 Subject: [PATCH 2/6] AMQP --- spec/AMQP/20-AMQP_FORMAT.md | 147 ++++++++++++++++++++------ spec/AMQP/21-AMQP_FORMAT_RATIONALE.md | 30 ++++++ 2 files changed, 146 insertions(+), 31 deletions(-) create mode 100644 spec/AMQP/21-AMQP_FORMAT_RATIONALE.md diff --git a/spec/AMQP/20-AMQP_FORMAT.md b/spec/AMQP/20-AMQP_FORMAT.md index 5c43ede2..97b5b61b 100644 --- a/spec/AMQP/20-AMQP_FORMAT.md +++ b/spec/AMQP/20-AMQP_FORMAT.md @@ -1,33 +1,118 @@ # AMQP format -The AMQP binding will define for traceparent and tracestate to be added to the -message in the application-properties section (as “w3c:tracestate” and -“w3c:traceparent”) if they are added by the original message publisher. Once the -message has been created, it’s no longer permissible to edit the bare message, -so if it were necessary to annotate the message inside the middleware as it -transits, that will happen in the “message-annotations” section, using the -“application-properties” as a base. The ultimate receiver can then pick between -the full context including infrastructure and the pure app context by either -looking into “message-annotations” or “application-properties”. - -1. *Protocol justification*. I suggest just reference W3C specs as they did in cloud events. And keep names of “headers”. -2. *Size-sensitive environments*. There is a good reason to keep the format and sizes unchanged. For the size-sensitive environments -If you ONLY INITIATE the message outside of existing context – there may be a possibility to save on size by truncating fields. -Discuss how to use binary protocol instead of textual. -Differences with message-id. Discuss how message-id is a deduplication concept where Trace-Context identifies the context of worker who either put or read the message. -Batching of messages – recommended semantics of TraceContext fields when batching is happening. - - -Failed post or get – what is relationship between failed operation trace - context and message trace context. -Log entire app properties to get correlation. - -Mutation of trace context by queue provider – should we allow/recommend to mutate trace context fields by trace provider. Especially for the span-id. - Security/privacy consideration – we can refer to W3C spec again. And adjust - it if needed. - -application properties is "unmodifiable" so "message-annotations" overrides -those in application - -traceparent 1.6.22 (list of binary) or binary - make rationale -tracestate 1.6.23 (map of string/string type) \ No newline at end of file +The Advanced Message Queuing Protocol (AMQP) is an open internet protocol for +business messaging. It defines a binary wire-level protocol that allows for the +reliable exchange of business messages between two parties. AMQP can be used as +a protocol for asynchronous communication between components of an application. +From the distributed tracing and telemetry correlation perspective it is a +known problem to be able to correlate a component that placed message and +component that processed it later. This specification describes how trace +context MUST be encoded into AMQP messages. + +## Trace context fields placement in a message + +AMQP defines message as a payload with the additional annotations sections. +There are two annotation sections this specification refers - +"application-properties" and "message-annotations". See +[3.2](http://docs.oasis-open.org/amqp/core/v1.0/os/amqp-core-messaging-v1.0-os.html#section-message-format) +of OASIS Advanced Message Queuing Protocol (AMQP) Version 1.0, Part 3: +Messaging. + +AMQP message section "application-properties" is immutable collection of +properties that is defined by message publisher and can be read by the message +consumer. Message brokers cannot mutate those properties. Section +"message-annotations" is designed for message brokers to use and can be mutated +during the message processing. + +Fields `traceparent` and `tracestate` SHOULD be added to the message in the +application-properties section by message publisher. Once the message has been +created, it’s no longer permissible to edit the bare message. So if it were +necessary to annotate the message inside the middleware as it transits, that +MUST happen in the “message-annotations” section, using the +“application-properties” as a base. Message reader SHOULD construct the full +trace context by reading `traceparent` and `tracestate` fields from the +“message-annotations” first and if not exist - from “application-properties”. + +## Trace context and failed read/write operations + +This specification defines how to propagate context from publisher thru broker +to ultimate reader. It does not define any additional transport level correlation +constructs that can be used to investigate failed publish or read operations. + +It is recommended, however, for AMQP implementations to make the best effort +attempt to read the trace context from the message and use it while reporting +such problems. + +## `traceparent` AMQP format + +The field `traceparent` MUST be encoded and decoded using [binary +protocol](..\extension-binary.html) and stored as a binary type defined in +section +[1.6.19](http://docs.oasis-open.org/amqp/core/v1.0/os/amqp-core-types-v1.0-os.html#type-binary) +of AMQP specification. + +Property name MUST be `traceparent` - all lowercase without delimiters. + +## `tracestate` AMQP format + +The field `tracestate` MUST be encoded and decoded as a string to string map. +See definition of type map in section +[1.6.23](http://docs.oasis-open.org/amqp/core/v1.0/os/amqp-core-types-v1.0-os.html#type-map) +of AMQP specification. + +--- + +*NOTE about the strings encoding* + +Please note that strings are defined as UTF-8 in AMQP. Should anything be +different from the HTTP encoded opaque strings? Should we allow to benefit from +wider character set for better encoding of opaque values? Or this will be too +error prone? + +--- + +Property name MUST be `tracestate` - all lowercase without delimiters. + +## Relations with message id field + +The field `message-id` is defined in section +[3.2.4](http://docs.oasis-open.org/amqp/core/v1.0/amqp-core-messaging-v1.0.html#type-properties) +of AMQP specification. + +Message-id, if set, uniquely identifies a message within the message system. The +message producer is usually responsible for setting the message-id in such a way +that it is assured to be globally unique. A broker MAY discard a message as a +duplicate if the value of the message-id matches that of a previously received +message sent to the same node. + +Trace context identifies the context of a worker who either publish or read the +message while message-id identifies the content of the message. So the message +with the same message-id can be send twice with the different values of trace +context fields in case of retries. Also trace context can be changed from broker +to broker to identify the broker while message-id will not be changed. + +It is recommended to use a unique `parent-id` for every publish of a message. So +it is not possible that `traceparent` will be identical for the messages with +the different `message-id`. + +## AMQP specific security and privacy considerations + +AMQP defines a protocol for a potentially long living messages. Long-term +storing of `traceparent` and `tracestate` fields may require additional handling +of security and privacy as that may not be covered by "in-flight" data +exemptions. + +So all the same privacy and security techniques should be applied with the +potentially more strict requirements. + +## Size sensitive environments + +TODO: There are many good reason to keep the format and sizes of fields +unchanged. For the size-sensitive environments, if you ONLY INITIATE the message +outside of existing context – there may be a possibility to save on size by +truncating or reusing some fields. + +## Batching + +TODO: Batching of messages – recommended semantics of TraceContext fields when +batching is happening. Should anything be said about it? \ No newline at end of file diff --git a/spec/AMQP/21-AMQP_FORMAT_RATIONALE.md b/spec/AMQP/21-AMQP_FORMAT_RATIONALE.md new file mode 100644 index 00000000..aca254c8 --- /dev/null +++ b/spec/AMQP/21-AMQP_FORMAT_RATIONALE.md @@ -0,0 +1,30 @@ +# Rationale for AMQP protocol decisions + +## Strings vs. complex types + +Complex types in most cases cheaper than string representation of `traceparent` +and `tracestate`. It is important in size restricted environments like IoT and +price-sensitive infrastructures where one pays for message metadata stored in a +queue server for a long time. + +Complex types are well defined and widely used in AMQP and they are part of a +protocol. So using them is fully supported by all existing clients. + +## Binary `traceparent` vs. list of binary values + +There are multiple ways to implement a `traceparent`. It can either be +implemented as string (http-like), binary protocol (like for grpc) or list of +separate binary values (`trace-id`, `parent-id`, `trace-flags`). + +Strings duplicating the size of a field, using list of binaries will require to +redefine the way the field serialized, parsed, and versioned. So re-using binary +protocol looks like a logical solution. + +## AMQP map for `tracestate` + +The benefit of using a built-in map type for AMQP is that serialization and +de-serialization of the field is built in and doesn't require any custom +implementation for parsing it. + +Maps in AMQP preserving the order so there is no conflict with the semantics of +the field. \ No newline at end of file From acbdfbb08ec6d2674c020a935036c2476e5fa500 Mon Sep 17 00:00:00 2001 From: Sergey Kanzhelev Date: Fri, 14 Dec 2018 13:39:09 -0800 Subject: [PATCH 3/6] prefix rationale --- spec/AMQP/21-AMQP_FORMAT_RATIONALE.md | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/spec/AMQP/21-AMQP_FORMAT_RATIONALE.md b/spec/AMQP/21-AMQP_FORMAT_RATIONALE.md index aca254c8..4399f112 100644 --- a/spec/AMQP/21-AMQP_FORMAT_RATIONALE.md +++ b/spec/AMQP/21-AMQP_FORMAT_RATIONALE.md @@ -27,4 +27,23 @@ de-serialization of the field is built in and doesn't require any custom implementation for parsing it. Maps in AMQP preserving the order so there is no conflict with the semantics of -the field. \ No newline at end of file +the field. + +## Prefix of the field names + +The properties are typically prefixed because they’re generic and might +interfere with an app’s use of the same names. So, for instance, instead of +`traceparent` prefixed name like `w3c:traceparent` can be used. In general AMQP +apps tend to be more metadata heavy than HTTP. For instance, `http:` is used in +the HTTP-over-AMQP spec. + +The question is whether prefix is required for trace context and if so - what +may be the name of the prefix. Options may be `w3c` for the origin of the spec, +`tcx` for trace context, `dt` for distributed tracing. + +However in the current spec, even if customer decided to use the same name, the +chance that customer didn’t mean to override those properties intentionally is +very small. Those are rare names. So prefix doesn’t add much, but increases a +chance for an error and interoperability. + +So suggestion is to keep the name un-prefixed. \ No newline at end of file From 7730f9d7ac497835a19f5a0a86f37865ca83cafa Mon Sep 17 00:00:00 2001 From: Sergey Kanzhelev Date: Wed, 2 Jan 2019 15:54:10 -0800 Subject: [PATCH 4/6] added rationale for app-props vs. message-annotations --- spec/AMQP/20-AMQP_FORMAT.md | 9 ++++++--- spec/AMQP/21-AMQP_FORMAT_RATIONALE.md | 9 +++++++++ 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/spec/AMQP/20-AMQP_FORMAT.md b/spec/AMQP/20-AMQP_FORMAT.md index 97b5b61b..b2c68ee1 100644 --- a/spec/AMQP/20-AMQP_FORMAT.md +++ b/spec/AMQP/20-AMQP_FORMAT.md @@ -15,8 +15,8 @@ AMQP defines message as a payload with the additional annotations sections. There are two annotation sections this specification refers - "application-properties" and "message-annotations". See [3.2](http://docs.oasis-open.org/amqp/core/v1.0/os/amqp-core-messaging-v1.0-os.html#section-message-format) -of OASIS Advanced Message Queuing Protocol (AMQP) Version 1.0, Part 3: -Messaging. +of *OASIS Advanced Message Queuing Protocol (AMQP) Version 1.0, Part 3: +Messaging* (AMQP specification). AMQP message section "application-properties" is immutable collection of properties that is defined by message publisher and can be read by the message @@ -43,6 +43,9 @@ It is recommended, however, for AMQP implementations to make the best effort attempt to read the trace context from the message and use it while reporting such problems. +TODO: do we need to define a way to specify trace context of read operation? Is +there anything in AMQP protocol that can be used to carry this context? + ## `traceparent` AMQP format The field `traceparent` MUST be encoded and decoded using [binary @@ -114,5 +117,5 @@ truncating or reusing some fields. ## Batching -TODO: Batching of messages – recommended semantics of TraceContext fields when +TODO: Batching of messages – recommended semantics of trace context fields when batching is happening. Should anything be said about it? \ No newline at end of file diff --git a/spec/AMQP/21-AMQP_FORMAT_RATIONALE.md b/spec/AMQP/21-AMQP_FORMAT_RATIONALE.md index 4399f112..5664efd0 100644 --- a/spec/AMQP/21-AMQP_FORMAT_RATIONALE.md +++ b/spec/AMQP/21-AMQP_FORMAT_RATIONALE.md @@ -29,6 +29,15 @@ implementation for parsing it. Maps in AMQP preserving the order so there is no conflict with the semantics of the field. +## Why use both - application and message properties + +Trace context is defined in an app in a context that mostly operates with the +application-properties collection. So ideally trace context should be set and +carried as part of application-properties collection. However, +application-properties are immutable so there should be a fallback mechanism to +use message-annotations if tracing must be implemented by one of message +brokers. + ## Prefix of the field names The properties are typically prefixed because they’re generic and might From bfe31959c66d9cfb70c0bdf5abd5b77cb6203cdc Mon Sep 17 00:00:00 2001 From: Sergey Kanzhelev Date: Mon, 28 Jan 2019 15:32:24 -0800 Subject: [PATCH 5/6] rationale for deliver-annotations --- spec/AMQP/21-AMQP_FORMAT_RATIONALE.md | 28 +++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/spec/AMQP/21-AMQP_FORMAT_RATIONALE.md b/spec/AMQP/21-AMQP_FORMAT_RATIONALE.md index 5664efd0..0ddde278 100644 --- a/spec/AMQP/21-AMQP_FORMAT_RATIONALE.md +++ b/spec/AMQP/21-AMQP_FORMAT_RATIONALE.md @@ -38,6 +38,34 @@ application-properties are immutable so there should be a fallback mechanism to use message-annotations if tracing must be implemented by one of message brokers. +## Why not delivery-annotations + +There are three grand scenarios: + +*Application trace context* through a simple AMQP brokered entity. If you send a +message into a queue from an app and then pull that message from the queue with +the same or another app and nothing else interesting happens, +application-properties is the right place, because you're not doing any tracing +of the broker, you're just passing the context alongside the message. + +*Application context plus routing context*. If you have a chain of AMQP entities +as you might have in a complex routing scenario, you might want to enrich the +context as the message is being passed on. In that case, the prior scenario +still exists, i.e. you will still want the option to have the context +information as set by the message producer and use that for application level +tracing unencumbered by whatever the routing layer might be tracking. As the +routing layer gets it hand on the message for tracing, it basically forks the +context off into the message-annotations and that's being manipulated on the +route. As the message reaches the destination, you now have the app-level +context from the producer in the application-properties AND the routing context +that was layered on top of that in the message-annotations. + +Hop to hop tracing: A delivery-annotations usage scenario would be purely +additive to either of the prior two, allowing any intermediary to spawn a new +context for itself or to propagate a context only via subset of hops. I think +that is fairly esoteric at this point and we should NOT take that into a spec +unless someone shows a hard use-case. + ## Prefix of the field names The properties are typically prefixed because they’re generic and might From 39cf9c367c1d34ce293425ad5e2ff560150053f3 Mon Sep 17 00:00:00 2001 From: Sergey Kanzhelev Date: Tue, 29 Jan 2019 13:02:23 -0800 Subject: [PATCH 6/6] message about batching --- spec/AMQP/20-AMQP_FORMAT.md | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/spec/AMQP/20-AMQP_FORMAT.md b/spec/AMQP/20-AMQP_FORMAT.md index b2c68ee1..b61ce608 100644 --- a/spec/AMQP/20-AMQP_FORMAT.md +++ b/spec/AMQP/20-AMQP_FORMAT.md @@ -117,5 +117,16 @@ truncating or reusing some fields. ## Batching -TODO: Batching of messages – recommended semantics of trace context fields when -batching is happening. Should anything be said about it? \ No newline at end of file +The specification defines a binding to the AMQP "message format 0", i.e. the +layout that is defined in the Messaging section of the the AMQP spec, but does +not provide any further detail on HOW those messages are being transferred. + +AMQP is enormously efficient when filling link credit with sending messages in +sequence without any special overlaid batching. And is often used without +batching. + +Message brokers like Azure Service Bus, Azure Event Hub and others have a +special message format that can batch multiple AMQP messages into one. However +the way to bind to that format is by specifying a binding to the individual +messages and then have the proprietary batching model pick that up via the +binding to the standard message. \ No newline at end of file