diff --git a/src/components/sidebar.tsx b/src/components/sidebar.tsx index c0aaaba22e..fe83170107 100644 --- a/src/components/sidebar.tsx +++ b/src/components/sidebar.tsx @@ -206,8 +206,8 @@ export default () => { Envelopes Rate Limiting - Trace Contexts Span Operations + Dynamic Sampling Context diff --git a/src/docs/sdk/performance/dynamic-sampling-context.mdx b/src/docs/sdk/performance/dynamic-sampling-context.mdx new file mode 100644 index 0000000000..217f8dbc63 --- /dev/null +++ b/src/docs/sdk/performance/dynamic-sampling-context.mdx @@ -0,0 +1,258 @@ +--- +title: "Dynamic Sampling Context (Experimental)" +--- + + + +This page is under active development. +Specifications are not final and subject to change. +Anything that sounds fishy probably is - nothing is set in stone. +Opening PRs to improve this page is therefore highly encouraged! + + + +Traces sampling done through the `tracesSampleRate` or `tracesSampler` options in the SDKs has quite a few consquences for users of Sentry SDKs: + +- Changing the sampling rate involved either redeploying applications (which is problematic in case of applications that are not updated automatically, i.e., mobile apps or physically distributed software) or building complex systems to dynamically fetch a sample rate. +- The sampling strategy leverages head-based simple random sampling. +- Employing sampling rules, for example, based on event parameters, is very complex. Sometimes this even requires users to have an in-depth understanding of the event schema. +- While writing rules for singular **transactions** is possible, enforcing them on entire **traces** is infeasible. + +The solution for these problems is **Dynamic Sampling**. +Dynamic Sampling allows users to configure **sampling rules** directly in the Sentry interface. Important: Sampling rules may be applied to **entire traces** or to a **single transaction**. + +## High-Level Problem Statement + +### Ingest + +Implementing Dynamic Sampling comes with challenges, especially on the ingestion side of things. +For Dynamic Sampling, we want to make sampling decisions for entire traces. +However, to keep ingestion speedy, Relay only looks at singular transactions in isolation (as opposed to looking at whole traces). +This means that we need the exact same decision basis for all transactions belonging to a trace. +In other words, all transactions of a trace need to hold all of the information to make a sampling decision, and that **information needs to be the same across all transactions of the trace**. +We call the information we base sampling decisions on **"Dynamic Sampling Context"** or **"DSC"**. + ++Currently, we can dynamically sample in two ways. First, we can do dynamic sampling on single transactions. For this process, Relay looks at the incoming event payload to make decisions. Second, we can do dynamic sampling across an entire trace. For this process, Relay relies on a **Dynamic Sampling Context**. + +As a mental model: +The head transaction in a trace determines the Dynamic Sampling Context for all following transactions in that trace. +No information can be changed, added or deleted after the first propagation. +Dynamic Sampling Context is bound to only one particular trace, and all the transactions that are part of this trace. +Multiple different traces can and should have different Dynamic Sampling Contexts. + +### SDKs + +SDKs are responsible for propagating **Dynamic Sampling Context** across all applications that are part of a trace. +This involves: + +1. Collecting the information that makes up the DSC **xor** extracting the DSC from incoming requests. +2. Propagating DSC to downstream SDKs. +3. Sending the DSC to Sentry via the `trace` envelope header. + +Because there are quite a few things to keep in mind for DSC propagation and to avoid every SDK running into the same problems, we defined a [unified propagation mechanism](#unified-propagation-mechanism) (step-by-step instructions) that all SDK implementations should be able to follow. + +## Baggage + +We chose `baggage` as the propagation mechanism for DSC. ([w3c baggage spec](https://www.w3.org/TR/baggage/)) +Baggage is a standard HTTP header with URI encoded key-value pairs. + +For the propagation of DSC, SDKs first read the DSC from the baggage header of incoming requests/messages. +To propagate DSC to downstream SDKs/services, we create a baggage header (or modify an existing one) through HTTP request instrumentation. + + + +Other vendors might also be using the `baggage` header. +If a `baggage` header already exists on an outgoing request, SDKs should aim to be good citizens by only **appending** Sentry values to the header. +In the case that another vendor added Sentry values to an outgoing request, SDKs may overwrite those values. + +SDKs must not add other vendors' baggage from incoming requests to outgoing requests. +Sentry SDKs only concern themselves with Sentry baggage. + + + +The following is an example of what a baggage header containing Dynamic Sampling Context may look like: + +``` +baggage: other-vendor-value-1=foo;bar;baz, sentry-trace_id=771a43a4192642f0b136d5159a501700, sentry-public_key=49d0f7386ad645858ae85020e393bef3, sentry-sample_rate=0.01337, sentry-user_id=Am%C3%A9lie, other-vendor-value-2=foo;bar; +``` + +See the [Payloads section](#payloads) for a complete list of key-value pairs that SDKs should propagate. + +## Freezing Dynamic Sampling Context + +As mentioned above, in order to be able to make sampling decisions for entire traces, Dynamic Sampling Context must be the same across all transactions of a trace. + +**What does this mean for SDKs?** + +When starting a new trace, SDKs are no longer allowed to alter the DSC for that trace as soon as this DSC leaves the boundaries of the SDK for the first time. +The DSC is then considered "frozen". +DSC leaves SDKs in two situations: + +- When an outgoing request with a `baggage` header, containing the DSC, is made. +- When a transaction envelope containing the DSC is sent to Sentry + +When an SDK receives an HTTP request that was "instrumented" or "traced" by a Sentry SDK, the receiving SDK should consider the incoming DSC as instantly frozen. +Any values on the DSC should be propagated "as is" - this includes values like "environment" or "release". + +SDKs should recognize incoming requests as "instrumented" or "traced" when at least one of the following applies: + +- The incoming request has a `sentry-trace` header +- The incoming request has a `baggage` header containing one or more keys starting with "`sentry-`" + +After the DSC of a particular trace has been frozen, API calls like `set_user` or `set_transaction` should have no effect on the DSC. + +## Payloads + +Dynamic Sampling Context is sent to Sentry via the `trace` envelope header and is propagated to downstream SDKs via a baggage header. + +All of the values in the payloads below are required (non-optional) in a sense, that when they are known to an SDK at the time a transaction envelope is sent to Sentry, or at the time a baggage header is propagated, they must also be included in said envelope or baggage. +In any case, `trace_id`, `public_key`, and `sample_rate` should always be known to an SDK, so these values are strictly required. + +### Envelope Header + +Dynamic Sampling Context is transferred to Sentry through the `trace` envelope header. +The value of this envelope header is a JSON object with the following fields: + +- `trace_id` (string) - The original trace ID as generated by the SDK, UUID V4 encoded as a hexadecimal sequence with no dashes (e.g. `771a43a4192642f0b136d5159a501700`) that is a sequence of 32 hexadecimal digits. This must match the trace id of the submitted transaction item. +- `public_key` (string) - Public key from the DSN used by the SDK. It allows Sentry to sample traces spanning multiple projects, by resolving the same set of rules based on the starting project. +- `sample_rate` (string) - The sample rate as defined by the user on the SDK. This string should always be a number between (and including) 0 and 1 in basic float notation (`0.04242`) - no funky business like exponents or anything similar. +- `release` (string) - The release name as specified in client options`. +- `environment` (string) - The environment name as specified in client options. +- `user_id` (string) - User ID as set by the user with `scope.set_user`. +- `user_segment` (string) - User segment as set by the user with `scope.set_user`. +- `transaction` (string) - The transaction name set on the scope. + +It's important to note that at the current moment, only `release`, `environment`, `user_id`, `user_segment`, and `transaction` are used by the product for dynamic sampling functionality. The rest of the context attributes, `trace_id`, `public_key`, and `sample_rate`, are used by Relay for internal decisions (like transaction sample rate smoothing). + +### Baggage-Header + +SDKs may use the following keys to set entries on `baggage` HTTP headers: + +- `sentry-trace_id` +- `sentry-public_key` +- `sentry-sample_rate` +- `sentry-release` +- `sentry-environment` +- `sentry-user_id` +- `sentry-user_segment` +- `sentry-transaction` + +SDKs must set all of the keys in the form of "`sentry-[name]`". +The prefix "`sentry-`" acts to identify key-value pairs set by Sentry SDKs. + +All of the keys are defined in a way so their value directly corresponds to one of the fields on the `trace` envelope header. +**This allows SDKs to put all of the sentry key-value pairs from the `baggage` directly onto the envelope header, after stripping away the `sentry-` prefix.** + +Being able to simply copy key-value pairs from the baggage header onto the `trace` envelope header gives us the flexibility to provide dedicated API methods to propagate additional values using Dynamic Sampling Context. +This, in return, allows users to define their own values in the Dynamic Sampling Context so they can sample by those in the Sentry interface. + +## Unified Propagation Mechanism + +SDKs should follow these steps for any incoming and outgoing requests (in python pseudo-code for illustrative purposes): + +```python +def collect_dynamic_sampling_context(): + # Placeholder function that collects as many values for Dynamic Sampling Context + # as possible and returns a dict + +def has_sentry_value_in_baggage_header(request): + # Placeholder function that returns True when there is at least one key-value pair in the baggage + # header of `request`, for which the key starts with "sentry-". Otherwise, it returns False. + +def on_incoming_request(request): + if request.has_header("sentry-trace") and (not request.has_header("baggage") or not has_sentry_value_in_baggage_header(request)): + # Request comes from an old SDK which doesn't support Dynamic Sampling Context yet + # --> we don't propagate baggage for this trace + current_transaction.dynamic_sampling_context_frozen = True + elif request.has_header("baggage") and has_sentry_value_in_baggage_header(request): + current_transaction.dynamic_sampling_context_frozen = True + current_transaction.dynamic_sampling_context = baggage_header_to_dict(request.headers.baggage) + +def on_outgoing_request(request): + if not current_transaction.dynamic_sampling_context_frozen: + current_transaction.dynamic_sampling_context_frozen = True + current_transaction.dynamic_sampling_context = merge_dicts(collect_dynamic_sampling_context(), current_transaction.dynamic_sampling_context) + + if not current_transaction.dynamic_sampling_context: + # Make sure there is at least an empty DSC set on transaction + # This is independent of whether it is locked or not + current_transaction.dynamic_sampling_context = {} + + if request.has_header("baggage"): + outgoing_baggage_dict = baggage_header_to_dict(request.headers.baggage) + merged_baggage_dict = merge_dicts(outgoing_baggage_dict, current_transaction.dynamic_sampling_context) + request.set_header("baggage", dict_to_baggage_header(merged_baggage_dict)) + else: + request.set_header("baggage", dict_to_baggage_header(current_transaction.dynamic_sampling_context)) +``` + +While there is no strict necessity for the `current_transaction.dynamic_sampling_context_frozen` flag yet, there is a future use case where we need it: +We might want users to be able to set Dynamic Sampling Context values themselves. +The flag becomes relevant after the first propagation, where Dynamic Sampling Context becomes immutable. +When users attempt to set DSC afterwards, our SDKs should make this operation a noop. + +## Considerations and Challenges + +This section details some open questions and considerations that need to be addressed for dynamic sampling and the usage of the baggage propogation mechanism. +These are not blockers to the adoption of the spec, but instead are here as context for future developments of the dynamic sampling product and spec. + +### The Temporal Problem + +Unlike `environment` or `release`, which should always be known to an SDK at initialization time, `user_id`, `user_segment`, and `transaction` (name) are only known after SDK initialization time. +This means that if a trace is propagated from a running transaction _BEFORE_ the user/transaction attributes are set, you'll get a portion of transactions in a trace that have different Dynamic Sampling Context than other portions, leading to _dynamic sampling across a trace_ not working as expected for users. + +Let's say we want to dynamically sample a browser application based on the `user_id`. +In a typical single page application (SPA), the user information has to be requested from some backend service before it can be set with `Sentry.setUser` on the frontend. + +Here's an example of that flow: + +- Page starts loading +- Sentry initializes and starts `pageload` transaction +- Page makes HTTP request to user service to get user (propogates sentry-trace/baggage to user service) + - user service continues trace by automatically creating sampling transaction + - user service pings database service (propogates sentry-trace/baggage to database service) + - database service continues trace by automatically creating sampling transaction +- Page gets data from user service, calls `Sentry.setUser` and sets `user_id` +- Page makes HTTP requests to service A, service B, and service C (propogates sentry-trace/baggage to services A, B and C) + - DSC is propogated with baggage to service A, service B, and service C, so 3 child transactions +- Page finishes loading, finishing `pageload` transaction, which is sent to Sentry + +In this case, the baggage that is propogated to the user service and the downstream database service _does not_ have the `user_id` value in it, because it was not yet set on the browser SDK. +Therefore, when Relay tries to dynamically sample the user services and database services transactions based on `user_id`, it will not be able to. +In addition, since the DSC is frozen after it's been sent, the DSC sent to service A, service B, and service C will not have `user_id` on it either. This means it also will not be dynamically sampled properly if there is a trace-wide DS rule on `user_id`. + +This problem exists for both `user_id` and `user_segment`, and it is because since we don't have the user information on some platforms/frameworks right as we initialize the SDK. + +For `transaction` name, the problem is similar, but it is because of paramaterization. +As much as we can, the SDKs will try to paramaterize transaction names (for ex, turn `/teams/123/user/456` into `/teams/:id/user/:id`) so that similar transactions are grouped together in the UI. +This improves both aggregate statistics for transactions and the general UX of using the product (setting alerts, checking measurements like web vitals, etc.). +For some frameworks, for example React Router v4 - v6, we paramaterize the transaction name after the transaction has been started due to constraints with the framework itself. +To illustrate this, let's look at another example: + +- Page starts loading +- Sentry initializes and starts `pageload` transaction (with transaction name `/teams/123/user/456` - based on window URL) +- Page makes HTTP request to service A (propogates sentry-trace/baggage to user service) +- Page renders with react router, triggering paramaterization of transaction name (`/teams/123/user/456` -> `/teams/:id/user/:id`). +- Page finishes loading, finishing pageload transaction, which is sent to Sentry + +When the pageload transaction shows up in Sentry, it'll be called `/teams/:id/user/:id`, but the Dynamic Sampling Context has `/teams/123/user/456` propogated, which means that the transaction from service A will not be affected by any trace-wide dynamic sampling rules put on the transaction name. +This will be very confusing to users, as effectively the transaction name they thinking that works will not. + +### Choosing Baggage as the Propogation Mechanism + +For more information on this, check the [DACI around using baggage](https://www.notion.so/sentry/Trace-Context-vs-Baggage-f541525012344111921b6aa7bfd78dc4) (Sentry employees only). + +Previously, we were using the [w3c Trace Context spec](https://www.w3.org/TR/trace-context/) as the mechanism to propogate dynamic sampling context between SDKs. +We switched to the [w3c Baggage spec](https://www.w3.org/TR/baggage) though because it was easier to use, required less code on the SDK side, and had much more [liberal size limits](https://www.w3.org/TR/baggage/#limits) than the trace context spec. +To make sure that we were not colliding with user defined keys as baggage is an open standard, we decided to prefix all sentry related keys with `sentry-` as a namespace. +This allowed us to use baggage as an open standard while only having to worry about Sentry data. + +### Other Items + +TODO - Add some sort of Q&A section on the following questions, after evaluating if they still need to be answered: + +- Why must baggage be immutable before the second transaction has been started? +- What are the consequences and impacts of the immutability of baggage on Dynamic Sampling UX? +- Why can't we just make the decision for the whole trace in Relay after the trace is complete? +- What is sample rate smoothing and how does it use `sample_rate` from the Dynamic Sampling Context? +- What are the differences between Dynamic Sampling on traces vs. transactions? diff --git a/src/docs/sdk/performance/trace-context.mdx b/src/docs/sdk/performance/trace-context.mdx deleted file mode 100644 index 1438cc93b8..0000000000 --- a/src/docs/sdk/performance/trace-context.mdx +++ /dev/null @@ -1,274 +0,0 @@ ---- -title: "Trace Contexts (Experimental)" ---- - - - -This feature is under development and not required for all SDKs supporting Performance Monitoring, yet. Please consider the Performance Guidelines as reference documentation. Anything that contradicts it is a mistake (or an out of date detail) in this document. - - - -In order to sample traces we need to pass along the call chain a trace id together with the necessary information for making a sampling decision, the so-called "trace context". - -## Protocol - -Trace information is passed between SDKs as an encoded `tracestate` header, which SDKs are expected to intercept and propagate. - -For event submission to sentry, the trace context is sent as JSON object embedded in an Envelope header with the key `trace`. - -### Trace Context - -Regardless of the transport mechanism, the trace context is a JSON object with the following fields: - -- `trace_id` (string, required) - UUID V4 encoded as a hexadecimal sequence with no dashes (e.g. `771a43a4192642f0b136d5159a501700`) that is a sequence of 32 hexadecimal digits. This must match the trace id of the submitted transaction item. -- `public_key` (string, required) - Public key from the DSN used by the SDK. It allows Sentry to sample traces spanning multiple projects, by resolving the same set of rules based on the starting project. -- `release` (string, optional) - The release name as specified in client options, usually: `package@x.y.z+build`. _This should match the `release` attribute of the transaction event payload_.\* -- `environment` - The environment name as specified in client options, for example `staging`. _This should match the `environment` attribute of the transaction event payload_.\* -- `user` (object, optional) - A subset of the scope's user context containing the following fields: - - `id` (string, optional) - The `id` attribute of the user context. - - `segment` (string, optional) - The value of a `segment` attribute in the user's data bag, if it exists. In the future, this field may be promoted to a proper attribute of the user context. -- `transaction` (string, optional) - The transaction name set on the scope. _This should match the `transaction` attribute of the transaction event payload_.\* - -\*_ See "Freezing the Context" for more information on consistency between the trace context and fields in the event payload._ - -**Example:** - -```json -{ - "trace_id": "771a43a4192642f0b136d5159a501700", - "public_key": "49d0f7386ad645858ae85020e393bef3", - "release": "myapp@1.1.2", - "environment": "production", - "user": { - "id": "7efa4978da177713df088f846f8c484d", - "segment": "vip" - }, - "transaction": "/api/0/project_details" -} -``` - -### Envelope Headers - -When sending transaction events to Sentry via Envelopes, the trace information must be set in the envelope headers under the `trace` field. - -Here's an example of a minimal envelope header containing the trace context (Although the header does not contain newlines, in the example below newlines were added for readability): - -```json -{ - "event_id": "12c2d058d58442709aa2eca08bf20986", - "trace": { - "trace_id": "771a43a4192642f0b136d5159a501700", - "public_key": "49d0f7386ad645858ae85020e393bef3" - // other trace attributes - } -} -``` - -### Tracestate Headers - -When propagating trace contexts to other SDKs, Sentry uses the [W3C `tracestate` header](https://www.w3.org/TR/trace-context/#trace-context-http-headers-format). See "Trace Propagation" for more information on how to propagate these headers to other SDKs. - -Tracestate headers contain several vendor-specific opaque data. As per HTTP spec, these multiple header values can be given in two ways, usually supported by HTTP libraries and framework out-of-the-box: - -- Joined by comma: - ``` - tracestate: sentry=,other= - ``` -- Repetition: - ``` - tracestate: sentry= - tracestate: other= - ``` - -To create contents of the tracestate header: - -1. Serialize the full trace context to JSON, including the trace_id. -2. Encode the resulting JSON string as UTF-8, if strings are represented differently on the platform. -3. Encode the UTF-8 string with base64. -4. Strip trailing padding characters (`=`), since this is a reserved character. -5. Prepend with `"sentry="`, resulting in `"sentry="`. -6. Join into the header as described above. - - - -By stripping trailing padding, default base64 parsers may detect an incomplete payload. Choose a parsing mode that either allows for missing `=` or allows truncated payloads. - - - -For example, the data - -```json -{ - "trace_id": "771a43a4192642f0b136d5159a501700", - "public_key": "49d0f7386ad645858ae85020e393bef3", - "release": "1.1.22", - "environment": "dev", - "user": { - "segment": "vip", - "id": "7efa4978da177713df088f846f8c484d" - } -} -``` - -would encode as - -``` -ewogICJ0cmFjZV9pZCI6ICI3NzFhNDNhNDE5MjY0MmYwYjEzNmQ1MTU5YTUwMTcwMCIsCiAgInB1YmxpY19rZXkiOiAiNDlkMGY3Mzg2YWQ2NDU4NThhZTg1MDIwZTM5M2JlZjMiLAogICJyZWxlYXNlIjogIjEuMS4yMiIsCiAgImVudmlyb25tZW50IjogImRldiIsCiAgInVzZXIiOiB7CiAgICAic2VnbWVudCI6ICJ2aXAiLAogICAgImlkIjogIjdlZmE0OTc4ZGExNzc3MTNkZjA4OGY4NDZmOGM0ODRkIgogIH0KfQ -``` - -and result in the header - -``` -tracestate: sentry=ewogIC...IH0KfQ,other=[omitted] -``` - -(Note the third-party entry at the end of the header; new or modified entries are always added to the lefthand side, so we put the `sentry=` value there. Also note that though the encoded value has been elided for legibilty here, in a real header the full value would be used.) - -## Implementation Guidelines - -An SDK supporting this header must: - -- Use scope information when creating a new trace context -- Add an envelope header with the trace context for envelopes containing transactions -- Add a `tracestate` HTTP header to outgoing HTTP requests for propagation -- Intercept incoming HTTP requests for `tracestate` HTTP headers where applicable and apply them to the local trace context - -### Background - -This is an extension of trace ID propagation covered by Performance Guidelines. According to the Unified API tracing spec, Sentry SDKs add an HTTP header `sentry-trace` to outgoing requests via integrations. Most importantly, this header contains the trace ID, which must match the trace id of the transaction event and also of the trace context below. - -The trace context shall be propagated in an additional `tracestate` header defined in [W3C traceparent header](https://www.w3.org/TR/trace-context/#trace-context-http-headers-format). Note that we must keep compatibility with the W3C spec as opposed to the proprietary `sentry-trace` header. The `tracestate` header also contains vendor-specific opaque data in addition to the contents placed by the Sentry SDK. - -### Client options - -While trace contexts are under development, they should be gated behind an internal `trace_sampling` boolean client option. The option defaults to `false` and should not be documented in Sentry docs. - -Based on platform naming guidelines, the option should be cased appropriately: - -- `trace_sampling` in snake case -- `traceSampling` in camel case -- `TraceSampling` in pascal case -- `setTraceSampling` for Java-style setters - -### Adding the Envelope Header - -The SDK should add the envelope header to outgoing envelopes under any of the following conditions: - -1. The envelope contains a transaction event. -2. The scope has a transaction bound. - -Specifically, this means even envelopes without transactions can contain the `trace` envelope header, -allowing Sentry to eventually sample attachments belonging to a transaction. When the envelope includes -a transaction and the scope has a bound transaction, the SDK should use the transaction of the envelope -to create the `trace` envelope header. - -### Freezing the Context - -To ensure fully consistent trace contexts for all transactions in a trace, the trace context cannot be changed once it is sent over the wire, even if scope or options change afterwards. That is, once computed the trace context is no longer updated. Even if the app calls `setRelease`, the old release remains in the context. - -To compensate for lazy calls to functions like `setTransaction` and `setUser`, the trace context can be thought to be in two states: _NEW_ and _SENT_ . Initially, the context is in the _NEW_ state and it is modifiable. Once sent for first time, it becomes _SENT_ and can no longer change. - -We recommend the trace context should be computed on-the-fly the first time it is needed in any of: - -- Creating an Envelope -- Propagation to an outgoing HTTP request - -The trace context must be retained until the user starts a new trace, at which point a new trace context must be computed by the SDK. - - - -It is recommended that SDKs log modifications of attributes that would result in trace context changes like `user.id` when the trace context is frozen, in order to simplify debugging of common dynamic sampling pitfalls. - - - -### Incoming Contexts - -Same as for intercepting trace IDs from inbound HTTP requests, SDKs should read `tracestate` headers and assume the Sentry trace context, if specified. Such a context is immediately frozen in the _SENT_ state and should no longer allow for modifications. - -## - -## Platform Specifics - -### Encoding in JavaScript - -As mentioned, we need to encode the JSON trace context using UTF-8 strings. JavaScript internally uses UTF16 so we need to work a bit to do the transformation. - -The basic idea is presented in [this](https://attacomsian.com/blog/javascript-base64-encode-decode) article (and in [other](https://stackoverflow.com/questions/30106476/using-javascripts-atob-to-decode-base64-doesnt-properly-decode-utf-8-strings) places). - -In short here's the function that converts a context into a base64 string that can be saved in `tracestate`. In the end we went with a simpler implementation, but the idea is the same: - -```jsx -// Compact form -function objToB64(obj) { - const utf16Json = JSON.stringify(obj); - const b64 = btoa( - encodeURIComponent(utf16Json).replace( - /%([0-9A-F]{2})/g, - function toSolidBytes(match, p1) { - return String.fromCharCode("0x" + p1); - } - ) - ); - const len = b64.length; - if (b64[len - 2] === "=") { - return b64.substr(0, len - 2); - } else if (b64[len - 1] === "=") { - return b64.substr(0, len - 1); - } - return b64; -} - -// Commented -function objToB64(obj) { - // object to JSON string - const utf16Json = JSON.stringify(obj); - // still utf16 string but with non ASCI escaped as UTF-8 numbers) - const encodedUtf8 = encodeURIComponent(utf16Json); - - // replace the escaped code points with utf16 - // in the first 256 code points (the most wierd part) - const b64 = btoa( - endcodedUtf8.replace(/%([0-9A-F]{2})/g, function toSolidBytes(match, p1) { - return String.fromCharCode("0x" + p1); - }) - ); - - // drop the '=' or '==' padding from base64 - const len = b64.length; - if (b64[len - 2] === "=") { - return b64.substr(0, len - 2); - } else if (b64[len - 1] === "=") { - return b64.substr(0, len - 1); - } - return b64; -} -// const test = {"x":"a-🙂-读写汉字 - 学中文"} -// objToB64(test) -// "eyJ4IjoiYS3wn5mCLeivu+WGmeaxieWtlyAtIOWtpuS4reaWhyJ9" -``` - -And here's the function that accepts a base64 string (with or without '=' padding) and returns an object - -```jsx -function b64ToObj(b64) { - utf16 = decodeURIComponent( - atob(b64) - .split("") - .map(function(c) { - return "%" + ("00" + c.charCodeAt(0).toString(16)).slice(-2); - }) - .join("") - ); - return JSON.parse(utf16); -} - -// b64ToObj("eyJ4IjoiYS3wn5mCLeivu+WGmeaxieWtlyAtIOWtpuS4reaWhyJ9") -// {"x":"a-🙂-读写汉字 - 学中文"} -``` - -## Base64 with Command Line Utils - -The GNU `base64` command line utility comes with a switch to wrap the encoded -string. This is not compatible with the `tracestate` header and should be -avoided. If the base64 implementations creates multiple lines, they must be -joined together.