From 774cec07adf1fad6aa2b3e9d3a383d23751954e7 Mon Sep 17 00:00:00 2001 From: Katie Byers Date: Wed, 29 Apr 2020 19:24:12 -0700 Subject: [PATCH 01/12] first draft of theory section, random other edits --- .../performance/distributed-tracing.md | 271 ++++++++++++++---- .../performance/performance-glossary.md | 131 --------- 2 files changed, 219 insertions(+), 183 deletions(-) delete mode 100644 src/collections/_documentation/performance/performance-glossary.md diff --git a/src/collections/_documentation/performance/distributed-tracing.md b/src/collections/_documentation/performance/distributed-tracing.md index 349d496be45ea..62598c0330f27 100644 --- a/src/collections/_documentation/performance/distributed-tracing.md +++ b/src/collections/_documentation/performance/distributed-tracing.md @@ -12,83 +12,249 @@ Sentry's Performance features are currently in beta. For more details about acce level="warning" %} -Enabling tracing data collection augments your existing error data to capture interactions among your software systems. This lets Sentry tell you valuable metrics about your software health like throughput and latency, as well as expose the impact of errors across multiple systems. Tracing makes Sentry a more complete monitoring solution to diagnose and measure your application health. +Enabling tracing allows you to augment your existing error data by capturing interactions between your software systems. This lets Sentry track valuable metrics about your software performance, like throughput and latency, as well as show the impact of errors across multiple systems. Tracing makes Sentry a more complete monitoring solution, allowing you to diagnose problems more easily and measure your application's overall health, by revealing insights such as: -## Tracing & Distributed Tracing +- What occurred for a specific error event, or issue +- What conditions are causing bottlenecks or latency issues in the application +- Which endpoints or operations consume the most time + +## What is Tracing? + +Applications typically consist of interconnected components (also called services). Let's take as an example a modern web application, composed of the following components, separated by network boundaries: + +- Front End (Single-Page Application) +- Backend (REST API) +- Task Queue +- Database Server +- Cron Job Scheduler + +Each of these components may be written in a different language on a different platform. Each one can be instrumented individually with a Sentry SDK (to capture error data or crash reports) but that doesn't tell the full story because each piece is considered separately. Tracing allows you to tie all of that data together; in our example, that means being able to follow a request from the front end to the backend and back, pulling in data from any background tasks or notification jobs that request creates. Not only does this allow you to correlate Sentry error reports, to see how an error in one service may have propagated to another, it also allows you to get much better insights into which services may be having a negative impact on your applications overall performance. + +Before learning how to enable tracing in your application, it helps to understand a few key terms and how they relate to one another. + +### Traces, Transactions, and Spans + +A **trace** represents the record of the entire operation you want to measure or track - like page load, an instance of a user completing some action in your application, or a cron job in your backend. When that trace includes work done in multiple services, such as those listed above, it's called a **distributed trace**. + +Each trace consists of one or more tree-like structures called **transactions**, the nodes of which are called **spans**. In most cases, each transaction represents a single instance of a service being called, and each span within that transaction represents that service performing a single unit of work, be it calling a function within that service or making a call to a different service. + +Since the transaction has a tree structure, top-level spans can themselves be broken down into smaller spans, mirroring the way that one function may call a number of other, smaller functions; this is expressed using the parent-child metaphor, so that every span may be the **parent span** to multiple other spans. Further, since all trees must have a single root, there is always one span representing the transaction itself, of which all other spans are descendants. + +To make all of this more concrete, let's consider our example web app again. + +### Example: Slow Page Load + +Suppose your web application is slow to load, and you'd like to know why. A lot has to happen in order for your app to first get to a usable state: multiple requests to your backend, likely some work - including calls to your database or to outside APis - done there before responses are returned, and processing by the browser in order to render all of the returned data into something meaningful to the user. So which part of that process is slowing things down? + +`[kmclb: In what follows I'm not worrying yet about list formatting/indenting, what's a header, what's italics masquerading as a header, etc. Open to opinions here, and also just note-to-self-ing that it needs to be thought about eventually.]` + +Let's say that in this simplified example, when a user loads the app in their browser, the following happens in each service: + + _Browser_ -Applications (for example, web applications) typically consist of interconnected components (also called services). For instance, let us assume a modern web application could be composed of the following components separated by the network boundaries: + - 1 request each for HTML, CSS, and JS + - 1 rendering task, which sets off 2 requests for JSON data -- Frontend (Single-Page Application) -- Backend -- Background Queue -- Notification Job + _Backend_ -Each of these components may be written in a different language on a different platform. Today, all of these components can be instrumented individually with Sentry SDKs to capture error data or crash reports whenever an event occurs in any one of them. + - 3 requests which just require serving static files (the HTML, CSS, and JS) + - 2 requests for JSON data, of which + - 1 requires a call to the database, and + - 1 requires a call to an external API and work to process the results before returning them to the front end -With tracing, we can follow the journey of the API endpoint requests from their source (for example, from the frontend), and instrument code paths as these requests traverse each component of the application. This journey is called a [**trace**]({%- link _documentation/performance/performance-glossary.md -%}#trace). Traces that cross between components, as in our web application example, are typically called **distributed traces**. + _Database Server_ + - 1 request which requires a query to check authentication and a query to get data + +(The external API is not listed because it's not yours, and you can't see inside of it.) + +In this example, the entire page-loading process, including all of the above, would be represented by a single **trace**, and that trace would consist of the following **transactions**: + + - 1 browser transaction (for page load) + - 5 backend transactions (one for each request) + - 1 database server transaction (for the single DB request) + +Each transaction would be broken down into **spans** as follows: + + _Browser Page-load Transaction_: 7 spans + - 1 root span representing the entire page load + - 1 span each (3 total) for the HTML, CSS, and JS requests + - 1 span for the rendering task, which itself contains + - 2 child spans, one for each JSON request + +Let's pause here to make an important point: Some (though not all) of the browser transaction spans we just listed have a direct correspondence to backend transactions we listed earlier. Specifically, each request _span_ in the browser transaction corresponds to a separate request _transaction_ in the backend. In this situation, when a span in one service gives rise to a transaction in a subsequent service, we call the original span a parent span to _both_ the transaction and its root span. + +In our example, every transaction other than the initial browser page-load transaction is the child of a span in another service, which means that every root span other than the browser transaction root has a parent span (albeit in a different service). In a fully-instrumented system (one in which every service has tracing enabled) this pattern will always hold true: _the only parentless span will be the root of the initial transaction, and of the remaining parented spans, every one will live in the same service as its parent, except for the root spans, whose parents will live in a previous service_. This is worth noting because it is the one way in which transactions are not perfect trees - their roots can (and mostly do) have parents. + +Now, for the sake of completeness, back to our spans: + + _Backend HTML/CSS/JS Request Transactions_: 2 spans each `[kmclb: if transactions can have no children, this might be only 1 span, the root]` + - 1 root span representing the entire request (child of a browser span) + - 1 span for serving the static file + + _Backend Request with DB Call Transaction_: 2 spans + - 1 root span representing the entire request (child of a browser span) + - 1 span for querying the database (parent of the database server transaction) + + _Backend Request with API Call Transaction_: 3 spans + - 1 root span representing the entire request (child of a browser span) + - 1 span for the API request (unlike with the DB call, _not_ a parent span, since the API isn't yours) + - 1 span for processing the API data + + _Database Server Request Transaction_: 3 spans + - 1 root span representing the entire request (child of the backend span above) + - 1 span for the authentication query + - 1 span for the query retrieving data + +To wrap up the example, after instrumenting all of your services, you might discover that - for some reason - of all things it's the auth query in your database server which is making things slow, accounting for more than half of the time it takes for your entire page load process to complete. Tracing can't tell you _why_ that's happening, but at least now you know where to look! + +`[kmclb: this def needs an illustration (or multiple illustrations) - can we maybe just adjust the current one for this example, or should we use one with more specifics? Current diagram:]` [{% asset performance/tracing-diagram.png alt="Diagram illustrating how a trace is composed of multiple transactions." %}]({% asset performance/tracing-diagram.png @path %}) + +### Further Examples -Each [trace]({%- link _documentation/performance/performance-glossary.md -%}#trace) has a marker called a `trace_id`. Trace IDs are pseudorandom fixed-length alphanumeric character sequences. +Here are a few more examples of potential usecases for tracing, broken down into transactions and spans: -By collecting traces of your users as they use your applications, you can begin to reveal some insights such as: +(Note: starred spans represent spans which are the parent of a later transaction (and its root span)) -- What occurred for a specific error event, or issue -- What conditions are causing bottlenecks or latency issues in the application -- Which endpoints or operations consume the most time +**Particular User Actions** -### Trace Propagation Model +If you application involves ecommerce, you might want to measure the time between a user clicking "Submit Order" and the order confirmation appearing, including tracking the submitting of the charge to the payment processor and the sending of an order confirmation email. Here that entire process would be one trace, and you'd likely have transactions for: -Let's refer back to our earlier example of the web application consisting of these high-level components: + - The browser's full process (example spans: XHR request to backend\*, rendering confirmation screen) + - Your backend's processing of that request (example spans: function call to compute total, DB call to store order*, API call to payment processor, queuing of email confirmation\*) + - Your database's work updating the customer's order history (example spans: individual SQL queries) + - The queued task of sending the email (example spans: function call to populate email template, API call to email-sending service) + +**Background Operations** -- Frontend (Single-Page Application) -- Backend -- Background Queue -- Notification Job +If your backend periodically polls for data from an external service, processes it, caches it, and then forwards it to an internal service, each instance of this happening would be a trace, and you'd likely have transactions for: -A trace [propagates]({%- link _documentation/performance/performance-glossary.md -%}#propagation) first from the frontend, then to the backend, and later to the background queue or notification job. Collected [spans]({%- link _documentation/performance/performance-glossary.md -%}#span) from each component are sent back to Sentry asynchronously and independently. Instrumented spans received from one component aren't forwarded to the next component. If a span appears to be missing a parent span, it could be an [**orphan span**]({%- link _documentation/performance/performance-glossary.md -%}#orphan-spans). +- The cron job which completes the entire process (example spans: API call to external service, processing function, call to caching service\*, API call to internal service\*) +- The work done in your caching service (example spans: checking cache for existing data, storing new data in cache) +- Your internal service's processing of the request (example spans: anything that service might do to handle the request) + +### Tracing Data Model -## Transactions +`[kmclb: do we want to format this in a quote-y kind of way?]` +"Show me your flowchart and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won't usually need your flowchart; it'll be obvious." -- [Fred Brooks](https://en.wikipedia.org/wiki/Fred_Brooks), The Mythical Man Month (1975) -[{% asset performance/anatomy-of-transaction.png alt="Diagram illustrating how a transaction is composed of many spans." %}]({% asset performance/anatomy-of-transaction.png @path %}) +All of the above theory is very nice, but traces, transactions, and spans are ultimately defined by what's in them, and the relationship among them is defined by how links between them are recorded. -Traces in Sentry are segmented into [spans]({%- link _documentation/performance/performance-glossary.md -%}#span) called [**transactions**]({%- link _documentation/performance/performance-glossary.md -%}#transaction). When instrumenting the application with tracing, collected spans are grouped within an encompassing top-level span called the **transaction span**. This notion of a transaction is specific to Sentry. +**Traces** -If you are collecting transactions from more than a single machine, you will likely encounter [**clock skew**]({%- link _documentation/performance/performance-glossary.md -%}#clock-skew). +Traces are not an entity in and of themselves. Rather, a trace is defined to be the collection of all transactions which share a `trace_id` value. -## Sampling Transactions +**Transactions** -**We strongly recommend sampling your transactions.** +Transactions share most of their properties (start and end time, tags, etc) with their root spans, so the same options are available (see below) and setting them in either place is equivalent. Transactions also have two extra properties not included in spans: + +`[kmclb: anything missing from this list?]` + +- `transaction_name`: used in various places in the UI to identify the transaction +- `root_span`: pointer to the root of the transactions span tree + +`[kmclb: how the heck do you get less-than or greater-than symbols to show up in code snippets?]` +Common examples of `transaction_name` values include endpoint paths (like `/store/checkout/` or `api/v2/users/\/`) for backend request transactions, task names (like `data.cleanup.delete_inactive_users`) for cron job transactions, and URLs (like `https://docs.sentry.io/performance/distributed-tracing/`) for page-load transactions. + +**Spans** + +The majority of the data in a transaction lives in the individual spans it contains. Span data includes: + +`[kmclb: the list below isn't currently ground truth (quite), but the ground is potentially shifting. Will update once API is settled.]` + + - `parent_span_id`: to tie the span to its parent span + - `op`: short code identifying the type of operation the span is measuring + - `start_timestamp`: when the span was opened + - `end_timestamp`: when the span was closed + - `description`: longer description of the span's operation, often specific to that instance (optional) + - `status`: short code indicating operation's status (optional) + - `tags`: key-value pairs holding additional data about the span (optional) + - `data`: arbitrarily-structured additional data about the span (optional) + +An example use of the `op` and `description` properties together is `op: sql.query` and `description: SELECT * FROM users WHERE last_active < DATE_SUB(CURRENT_DATE, INTERVAL 1 YEAR)`. The `status` property is often used to indicate the success or failure of the span's operation (or for a response code in the case of HTTP requests). Finally, `tags` and `data` allow you attach further contextual information to the span, such as `function: middleware.auth.is_authenticated` for a function call or `request: {url: ..., headers: ... , body: ...}` for an http request. + +### Good to Know (needs a better name) + +A few more important points about traces, transactions, and spans, and they way they relate to one another: + +**Trace Duration**: Because a trace is just a collection of transactions, traces don't have their own start and end times. Instead, a trace is considered to begin when its earliest transaction starts, and ends when its latest transaction ends. This also means you can't explicitly start or end a trace, only the transactions in that trace. + +**Async Transactions**: Because of the possibility of asynchronous processes, child transactions may outlive the transactions containing their parent spans, sometimes by many orders of magnitude. For example, if an backend API call sets off a long-running processing task and then immediately returns a response, the backend transaction will finish (and its data will be sent to Sentry) long before the async task transaction does. Asynchronicity also means that the order in which transactions are sent to (and received by) Sentry is in no way dependent on the order in which they were created. (Because of the variability of transmission times, order of receipt for transactions in the same trace is also only somewhat correlated with order of completion.) + +**Orphan Transactions**: In theory, in a fully instrumented system, each trace should only contain one transaction and one span (the transaction's root) without a parent, namely the transaction in the originating service. However, in practice, you may not have tracing enabled in every one of your services, or an instrumented service may fail to report a transaction due to network disruption or other unforeseen circumstances. When this happens, you can end up with gaps in your trace hierarchy, and specifically, you may see transactions partway through the trace whose parent spans haven't been recorded as part of any known transactions. Such non-originating, parentless transactions are called **orphan transactions**. + +`[kmclb: depending on how we end up formatting this, may need to make this one paragraph]` +**Clock Skew**: If you are collecting transactions from multiple machines, you will likely encounter **clock skew**, which is the situation in which timestamps in one transaction don't align with timestamps in another. For example, if your backend makes a database call, the backend transaction logically should start before the database transaction does. But if the system time on the two machines (hosting your backend and database, respectively) aren't synced to common standard, it's possible that won't be the case. It's also possible for the ordering to be correct but or the two recorded timeframes nonetheless to not line up in a way which is accurate to what actually happened. + +Because there is no way for Sentry to judge either the relative or absolute accuracy of your timestamps, it does not attempt to correct or modify them in any way. And while you can reduce clock skew by using Network Time Protocol (NTP) or your cloud provider's clock synchronization services, you may still notice small drifts in your data, as synchronizing clocks on small intervals is challenging. + +**Nesting Spans**: Though our examples above had four levels in their hierarchy (trace, transaction, span, child span) there's no set limit to how deep the nesting of spans can go. There are, however, practical limits: transaction payloads sent to Sentry have a maximum allowed size (currently `[kmclb: find out what this is]`), and there's a balance to be struck between your data's granularity and its usability. + +**Zero-duration Spans**: It's possible for a span to have equal start and end times, and therefore be recorded as taking no time. This can occur either because the span is being used as a marker (such as is done in [the browser's Performance API](https://developer.mozilla.org/en-US/docs/Web/API/Performance/mark)) or because the amount of time the operation took is less than the measurement resolution (which will vary by service). + +**What We Send**: Individual spans aren't sent to Sentry. Rather, those spans are attached to their containing transaction, and the transaction is sent as one unit. This means that no span data will be recorded by Sentry's servers until the transaction to which they belong is closed and dispatched. + +`[kmclb: STOP READING AT THIS POINT. Or, keep reding if you want, but I haven't worked on anything below here in a concerted way yet, so for the purposes of this initial feedback session, I'm really just looking for comments on ^^^^^^^^^]` + +## Data Sampling + +**When collecting traces, we strongly recommend sampling your data.** When you enable sampling for transaction events in Sentry, you choose a percentage of collected transactions to send to Sentry. For example, if you had an endpoint that received 1000 requests per minute, a sampling rate of 0.25 would result in 250 transactions (25%) being sent to Sentry. Sampling enables you to collect traces on a subset of your traffic and extrapolate to the total volume. Furthermore, **enabling sampling allows you to control the volume of data you send to Sentry and lets you better manage your costs**. If you don't have a good understanding of what sampling rate to choose, we recommend you start with a low value and gradually increase the sampling rate as you learn more about your traffic patterns and volume. -When you have multiple projects collecting transaction events, Sentry utilizes "head-based" sampling to ensure that once a sampling decision has been made at the beginning of the trace (typically the initial transaction), that decision is propagated to each service or project involved in the [trace]({%- link _documentation/performance/performance-glossary.md -%}#trace). If your services have multiple entry points, you should aim to choose a consistent sampling rate for each. Choosing different sampling rates can bias your results. Sentry does not support "tail-based" sampling at this time. +When you have multiple projects collecting transaction events, Sentry utilizes "head-based" sampling to ensure that once a sampling decision has been made at the beginning of the trace (typically the initial transaction), that decision is propagated to each service or project involved in the trace. If your services have multiple entry points, you should aim to choose a consistent sampling rate for each, as choosing different sampling rates can bias your results. Sentry does not support "tail-based" sampling at this time. If you enable Performance collection for a large portion of your traffic, you may exceed your organization's [Quotas and Rate Limits]({%- link _documentation/accounts/quotas/index.md -%}). -## Viewing Transactions +## Viewing Trace Data -View transaction events by clicking on the "Transactions" pre-built query in [Discover]({%- link _documentation/performance/discover/index.md -%}), or by using a search condition `event.type:transaction` in a [Discover Query Builder]({%- link _documentation/performance/discover/query-builder.md -%}) view. +You can see a list of transaction events by clicking on the "Transactions" pre-built query in [Discover]({%- link _documentation/performance/discover/index.md -%}), or by using a search condition `event.type:transaction` in the [Discover Query Builder]({%- link _documentation/performance/discover/query-builder.md -%}) view. -When you open a transaction event in Discover, you'll see the **span view** at the top of the page. Other information the SDK captured as part of the Transaction event, such as breadcrumbs, will also be displayed. +`[kmclb: I think we should have a section here on searching, which would mention that transactions (but not spans) are searchable, and what data one can use to search. Should also probably link to the main search docs.]` -[{% asset performance/discover-span.png alt="Discover span showing the map of the transactions (aka minimap) and the black dashed handlebars for adjusting the window selection." %}]({% asset performance/discover-span.png @path %}) +### Performance Metrics + +`[kmclb: we don't actually tell you *how* to do this...]` +Some aggregate functions can be applied to transaction events to help you better understand the performance characteristics of your applications. + +**Duration Percentiles** + + +You can aggregate transaction durations using the following functions: +- average +- 75%, 95%, and 99% duration percentiles + +One use case for tracking these statistics is to help you identify transactions that are slower than your organization's target SLAs. -### Using the Span View +A word of caution when looking at averages and percentiles: In most cases, you'll want to set up tracing so that only a fraction of possible traces are actually sent to Sentry, to avoid overwhelming your system. `[kmclb: include a link to sampling section]` Further, you may want to filter your transaction data by date or other factors, or you may be tracing a relatively uncommon operation. For all of these reasons, you may end up with average and percentile data that is directionally correct, but not accurate. (To use the most extreme case as an example, if only a single transaction matches your filters, you can still compute an "average" duration, even though that's clearly not what is usually meant by "average.") -Note that [traces]({%- link _documentation/performance/performance-glossary.md -%}#trace) are segmented into [spans]({%- link _documentation/performance/performance-glossary.md -%}#span) called [transactions]({%- link _documentation/performance/performance-glossary.md -%}#transaction). The span view enables you to examine the waterfall graph (or hierarchy) of the instrumented transaction. +`[kmclb: "We can calculate an average with less data than a 95th percentile" - is there some threshold below which we won't do the calculation?]` + +The problem of small sample size (and the resulting inability to be usefully accurate) will happen more often in some columns than others. `[kmclb: which ones?]` We can calculate an average with less data than it takes to calculate the 95th percentile, but it’ll also vary by row. (For example, `/settings/` will always get more traffic than `/settings/country/belgium/tax`.) + +**Requests Per Minute (RPM)** + +Requests Per Minute is a way to measure throughput. It is the average of all request durations, bucketed by the minute `[kmclb: start time or end time?]` for the current time window and query string. + + +### Transaction Detail View + +When you open a transaction event in Discover (by clicking on the ), you'll see the **span view** at the top of the page. Other information the SDK captured as part of the transaction event, such as the transaction's tags and automatically collected breadcrumbs, is displayed underneath and to the right of the span view. + +[{% asset performance/discover-span.png alt="Discover span showing the map of the transactions (aka minimap) and the black dashed handlebars for adjusting the window selection." %}]({% asset performance/discover-span.png @path %}) -The span view is a split view where the left-hand side is the tree view displaying the parent-child relationship of the spans, and the right-hand side displays spans represented as colored rectangles. Within the tree view (left-hand side), Sentry identifies spans by their **operation name** and their **description**. If you don't provide the description, Sentry uses the span id as the fallback. +#### Using the Span View -At the top of the span view is a minimap, which is a "map" of the transaction. It helps orient you to the specific portion of the transaction that you're viewing. +The span view is a split view where the left-hand side shows the transaction's span tree, and the right-hand side represents each span as a colored rectangle. Within the tree view, Sentry identifies spans by their `op` and `description` values. If a span doesn't have a description, Sentry uses the span's id as a fallback. The first span listed is always the transaction's root span, from which all other spans in the transaction descend. -The first top-level span is the transaction span, which encompasses all other spans within the transaction. +At the top of the span view is a minimap, which shows which specific portion of the transaction you're viewing. **Zooming In on a Transaction** -As displayed in the Discover span screenshot above, you can click and drag your mouse cursor across the minimap (top of the span view). You can also adjust the window selection by dragging the handlebars (black dashed lines). +As shown in the Discover span screenshot above, you can click and drag your mouse cursor across the minimap (top of the span view). You can also adjust the window selection by dragging the handlebars (black dashed lines). **Missing Instrumentation** @@ -102,7 +268,7 @@ Click on a row to expand the details of the span. From here, you can see all att **Search by Trace ID** -You can search using `trace id` by expanding any of the span details and click on "Search by Trace". +You can search using `trace id` by expanding any of the span details and clicking on "Search by Trace". You need **project permissions for each project** to see all the transactions within a trace. Each transaction in a trace is likely a part of a different project. If you don't have project permissions, some transactions may not display as part of a trace. @@ -110,7 +276,7 @@ You need **project permissions for each project** to see all the transactions wi Child transactions are shown based on your project permissions -- which are necessary to viewing transaction events. To check project permissions, navigate to **Settings >> [your organization] >> [your project] >> General**. -Some spans within a transaction may be the parent of another transaction. If you expand the span details, you may see the "View Child" button, which, when clicked, will lead to another transaction's details view. +Some spans within a transaction may be the parent of another transaction. When you expand the span details for such spans, you'll see the "View Child" button, which, when clicked, will lead to the child transaction's details view. ## Setting Up Tracing @@ -151,7 +317,7 @@ Many integrations for popular frameworks automatically capture traces. If you al - AIOHTTP web apps - Redis Queue (RQ) -[Spans]({%- link _documentation/performance/performance-glossary.md -%}#span) are instrumented for the following operations within a [transaction]({%- link _documentation/performance/performance-glossary.md -%}#transaction): +Spans are instrumented for the following operations within a transaction: - Database queries that use SQLAlchemy or the Django ORM - HTTP requests made with `requests` or the `stdlib` @@ -171,7 +337,7 @@ sentry_sdk.init( **Manual Instrumentation** -To manually instrument certain regions of your code, you can create a [transaction]({%- link _documentation/performance/performance-glossary.md -%}#transaction) to capture them. +To manually instrument certain regions of your code, you can create a transaction to capture them. The following example creates a transaction for a scope that contains an expensive operation (for example, `process_item`), and sends the result to Sentry: @@ -188,7 +354,7 @@ while True: **Adding More Spans to the Transaction** -The next example contains the implementation of the hypothetical `process_item` function called from the code snippet in the previous section. Our SDK can determine if there is currently an open `transaction` and add all newly created [spans]({%- link _documentation/performance/performance-glossary.md -%}#span) as child operations to the `transaction`. Keep in mind that each individual span also needs to be manually finished; otherwise, spans will not show up in the `transaction`. +The next example contains the implementation of the hypothetical `process_item` function called from the code snippet in the previous section. Our SDK can determine if there is currently an open transaction and add all newly created spans as child operations to that transaction. Keep in mind that each individual span also needs to be manually finished; otherwise, spans will not show up in the transaction. You can choose the value of `op` and `description`. @@ -232,7 +398,7 @@ Sentry.init({ To send traces, you will need to set the `tracesSampleRate` to a nonzero value. The configuration above will capture 25% of your transactions. -You can pass many different options to the Tracing integration (as an object of the form `{optionName: value}`), but it comes with reasonable defaults out of the box. It will automatically capture a [transaction]({%- link _documentation/performance/performance-glossary.md -%}#transaction) for every page load. Within that transaction, [spans]({%- link _documentation/performance/performance-glossary.md -%}#span) are instrumented for the following operations: +You can pass many different options to the Tracing integration (as an object of the form `{optionName: value}`), but it comes with reasonable defaults out of the box. It will automatically capture a trace for every page load. Within that transaction, spans are instrumented for the following operations: - XHR/fetch requests - If available: Browser Resources (Scripts, CSS, Images ...) @@ -250,11 +416,12 @@ The default value of `tracingOrigins` is `['localhost', /^\//]`. The JavaScript - Therefore, the option needs to be configured like this: `new ApmIntegrations.Tracing({tracingOrigins: ['api.example.com']})` - Now outgoing XHR/fetch requests to `api.example.com` will get the `sentry-trace` header attached +`[kmclb: does this only apply to web servers, or all services which accept incoming requests?]` *NOTE:* You need to make sure your web server CORS is configured to allow the `sentry-trace` header. The configuration might look like `"Access-Control-Allow-Headers: sentry-trace"`, but this depends a lot on your set up. If you do not whitelist the `sentry-trace` header, the request might be blocked. **Using Tracing Integration for Manual Instrumentation** -The tracing integration will create a [transaction]({%- link _documentation/performance/performance-glossary.md -%}#transaction) on page load by default; all [spans]({%- link _documentation/performance/performance-glossary.md -%}#span) that are created will be attached to it. Also, the integration will finish the transaction after the default timeout of 500ms of inactivity. The page is considered inactive if there are no pending XHR/fetch requests. If you want to extend the transaction's lifetime beyond 500ms, you can do so by adding more spans to the transaction. The following is an example of how you could profile a React component: +The tracing integration will create a transaction on page load by default; all spans that are created will be attached to it. Also, the integration will finish the transaction after the default timeout of 500ms of inactivity. The page is considered inactive if there are no pending XHR/fetch requests. If you want to extend the transaction's lifetime, you can do so by adding more spans to it. The following is an example of how you could profile a React component: ```javascript // This line starts an activity (and creates a span). @@ -305,7 +472,7 @@ shopCheckout() { } ``` -This example will send a [transaction]({%- link _documentation/performance/performance-glossary.md -%}#transaction) `shopCheckout` to Sentry, containing all outgoing requests that happen in `validateShoppingCartOnServer` as spans. The transaction will also contain a `task` span that measures how long `processAndValidateShoppingCart` took. Finally, the call to `ApmIntegrations.Tracing.finshIdleTransaction()` will finish the [transaction]({%- link _documentation/performance/performance-glossary.md -%}#transaction) and send it to Sentry. Calling this is optional; if it is not called, the integration will automatically send the [transaction]({%- link _documentation/performance/performance-glossary.md -%}#transaction) itself after the defined `idleTimeout` (default 500ms). +This example will send a transaction `shopCheckout` to Sentry, containing all outgoing requests that happen in `validateShoppingCartOnServer` as spans. The transaction will also contain a `task` span that measures how long `processAndValidateShoppingCart` took. Finally, the call to `ApmIntegrations.Tracing.finishIdleTransaction()` will finish the transaction and send it to Sentry. Calling this is optional; if it is not called, the integration will automatically send the transaction itself after the defined `idleTimeout` (default 500ms). **What is an activity?** @@ -313,7 +480,7 @@ The concept of an activity only exists in the `Tracing` integration in JavaScrip **Manual Transactions** -To manually instrument a certain region of your code, you can create a [transaction]({%- link _documentation/performance/performance-glossary.md -%}#transaction) to capture it. +To manually instrument a certain region of your code, you can create a transaction to capture it. This is valid for both JavaScript Browser and Node and works independent of the `Tracing` integration. The following example creates a transaction for a part of the code that contains an expensive operation (for example, `processItem`), and sends the result to Sentry: @@ -333,7 +500,7 @@ processItem(item).then(() => { **Adding Additional Spans to the Transaction** -The next example contains the implementation of the hypothetical `processItem ` function called from the code snippet in the previous section. Our SDK can determine if there is currently an open transaction and add to it all newly created [spans]({%- link _documentation/performance/performance-glossary.md -%}#span) as child operations. Keep in mind that each individual span needs to be manually finished; otherwise, that span will not show up in the transaction. +The next example contains the implementation of the hypothetical `processItem ` function called from the code snippet in the previous section. Our SDK can determine if there is currently an open transaction and add to it all newly created spans as child operations. Keep in mind that each individual span needs to be manually finished; otherwise, that span will not show up in the transaction. ```javascript function processItem(item, transaction) { @@ -408,11 +575,11 @@ app.use(Sentry.Handlers.errorHandler()); app.listen(3000); ``` -[Spans]({%- link _documentation/performance/performance-glossary.md -%}#span) are instrumented for the following operations within a [transaction]({%- link _documentation/performance/performance-glossary.md -%}#transaction): +Spans are instrumented for the following operations within a transaction: - HTTP requests made with `request` - `get` calls using native `http` and `https` modules -- Middlewares (Express.js only) +- Middleware (Express.js only) To enable this: @@ -436,7 +603,7 @@ Sentry.init({ **Manual Transactions** -To manually instrument a certain region of your code, you can create a [transaction]({%- link _documentation/performance/performance-glossary.md -%}#transaction) to capture it. +To manually instrument a certain region of your code, you can create a transaction to capture it. The following example creates a transaction for a part of the code that contains an expensive operation (for example, `processItem`), and sends the result to Sentry: @@ -458,7 +625,7 @@ app.use(function processItems(req, res, next) { **Adding Additional Spans to the Transaction** -The next example contains the implementation of the hypothetical `processItem ` function called from the code snippet in the previous section. Our SDK can determine if there is currently an open transaction and add to it all newly created [spans]({%- link _documentation/performance/performance-glossary.md -%}#span) as child operations. Keep in mind that each individual span needs to be manually finished; otherwise, that span will not show up in the transaction. +The next example contains the implementation of the hypothetical `processItem ` function called from the code snippet in the previous section. Our SDK can determine if there is currently an open transaction and add to it all newly created spans as child operations. Keep in mind that each individual span needs to be manually finished; otherwise, that span will not show up in the transaction. ```javascript function processItem(item, transaction) { diff --git a/src/collections/_documentation/performance/performance-glossary.md b/src/collections/_documentation/performance/performance-glossary.md deleted file mode 100644 index 773a98b84f818..0000000000000 --- a/src/collections/_documentation/performance/performance-glossary.md +++ /dev/null @@ -1,131 +0,0 @@ ---- -title: Performance Glossary -sidebar_order: 5 ---- - -## Clock Skew - -If you are collecting transactions from multiple machines, you will likely encounter **clock skew**. When collecting and comparing transaction events that are gathered from different devices, you may observe timestamps that do not align or events in a trace that don't share a time history. Sentry does not attempt to correct the timestamps in your events. The timestamps displayed in Sentry for transactions and each span retain the original values reported by your application/hosting environment. - -While you can reduce clock skew by utilizing Network Time Protocol (NTP) or your cloud provider's clock synchronization services, you may still notice small drifts in your data, as synchronizing clocks on small intervals is challenging. - -## Performance Metrics - -Some aggregate functions can be applied to transaction events to help you better understand the performance characteristics of your applications. - -### Duration Percentiles - -You can aggregate transaction durations using the following aggregate functions: -- average -- 75%, 95%, and 99% duration percentiles - -One use-case for using these statistics is to help you identify transactions that are slower than your organization's target SLAs. - -Transaction events are sampled. As a result, the percentiles presented within Sentry represent only the data received. Because of sampling, data ranges, filters, or low volume of transactions, we will often have a case where data is directionally correct, but not accurate. For example, the average of one number is that number, but that’s not usually what you may expect to see. - -This inability to be usefully accurate will happen more often in some columns than others. We can calculate an average with less data than a 95th percentile, but it’ll also vary by row. An index page will always get more traffic than `/settings/country/belgium/tax`. - -### Requests Per Minute (RPM) - -Requests Per Minute is a way to measure throughput. It is the average of all requests bucketed by the minute for the current time window and query string. - -## Span - -The **span** is the primary building block of a distributed trace, representing an individual unit of work done in a distributed system. It may represent the processing of an HTTP request by a server, the querying of a database, or an HTTP call to an external service. - -When you initialize and load a Sentry SDK that supports tracing, it will automatically instrument parts of your running application, such as function calls and network operations. Instrumenting a database call could entail these three steps or subroutines: - -1. Calling a function to compose a database query -2. Sending a query to a database -3. Receiving and transforming results - -Each of these subroutines takes some time to perform (no matter how fast they could run). Sentry records these subroutines as a unit of work in the form of **spans**. Sentry can also attach useful information to spans, such as tags, additional contextual data, and a status to indicate if the subroutine failed or not. - -Spans can have descendant spans (or child spans). In our earlier example, the three subroutines can be spans within a larger encompassing span (the database call). - -Any instrumented span is part of a trace (identified by its trace id, `trace_id`), and each span has its own marker called the `span_id`, a fixed length of alphanumeric characters. - -An example of a span that describes a call to an external service: - -```javascript -// updateJiraTask -{ - "trace_id": "a8233eb845154a2cae4712b28375d08d", - "parent_span_id": "ba99c37bfb30f860", - "span_id": "92ce7ea47d5b8a45", - "start_timestamp": 1583868931.940406, - "same_process_as_parent": true, - "description": "updateJiraTask", - "tags": { "status": "ok" }, - "timestamp": 1583868932.060615, - "op": "jira", - "data": {} -} - -// sql.update - a child span of updateJiraTask -{ - "trace_id": "a8233eb845154a2cae4712b28375d08d", - "parent_span_id": "92ce7ea47d5b8a45", - "span_id": "a88e33dab5b745b9", - "start_timestamp": 1583868931.953386, - "same_process_as_parent": true, - "description": "sql.update - SELECT * FROM jira_table", - "tags": { "status": "ok" }, - "timestamp": 1583868932.03692, - "op": "db", - "data": {} -} -``` - -### Hierarchy - -Each span can be connected to other spans using a parent-child hierarchy. - -For example, a span called `updateJiraTask` would have child spans like `Check permissions`, `Update Task`, `sql.update`, `send.webhooks`. - -Each sub-task span would have `parent_span_id` equal to the id of the `updateJiraTask` span. - -### Orphan Spans - -Orphan spans are spans that lack visible parent spans within Sentry's database. This could imply a bug in the Sentry application. If you're confident about this, please contact support: [support@sentry.io](mailto:support@sentry.io) - -Another reason could include that instrumented spans may not make it back to Sentry for processing. As an example, let us consider a web application consisting of: - -- Frontend (Single-Page Application) -- Backend -- Background Queue -- Notification Job - -Instrumented spans in each component are sent back to Sentry for processing. However, a server in the backend component could suffer from network connectivity loss, and the spans may not have made it back to Sentry. This would indicate a "hole" within this specific trace. The collected spans within the Background Queue would be descendants of a span (for example, a queue service call) within the Backend component, and thus the top-level span of the collective Background Queue spans is referred to as an orphan span. - -### Properties - -Each span has a unique identifier kept in the `span_id` attribute. - -Start time and end time are stored in the `start_timestamp` and `timestamp` attributes, respectively. The format is an RFC 3339 / ISO 8601-compatible string or a numeric value with the number of seconds since the UNIX Epoch. - -The name of the operation is stored in the `op` property. Example `op` values are `http`, `sql.query`, `redis.command`. - -Additional data about the operation can be stored in the `data` and `tags` attributes. - -## Trace - -A trace is a collection of transactions that have the same `trace_id` attribute. Traces consist of all transactions generated by the distributed system during the processing of the input data. Traces are not explicitly started or finished. The start and end of a trace are defined by the transactions and spans participating in the trace. - -Traces that start with the same set of API endpoint calls can potentially differ in terms of what code paths were executed. For example, an initial trace may result in a cache miss at the Cache Service, and a subsequent trace may include more information instrumented at the Cache Service. Although these two traces are similar, they are not identical. - -### Propagation - -Processing of the input data in the distributed environment means that an arbitrary number of the systems/microservices can be involved. - -To connect all the spans that were created during the processing of the input data, we need to be able to propagate the basic identifier. By forwarding `trace_id` and `parent_span_id` as data travels through various applications in a microservice ecosystem, we can reconstruct data flows later. - -Trace metadata is propagated through HTTP headers that are included in the HTTP request to the external system. - -## Transaction - -Transaction is an event attribute that provides initial context to the error report. This makes it possible for the error to be linked to a specific part of the system. An example of the transaction name can be “/users/” which is the name for the endpoint that received the request that failed during the processing. - -With tracing, the transaction is a parent span to all spans created during the processing of the request in the single server. - -Once the request is processed, the transaction is sent together with all child spans to Sentry. From d4eac653a1194bfded9fc276d0505b047f6d9c92 Mon Sep 17 00:00:00 2001 From: Fiona Artiaga Date: Fri, 1 May 2020 10:36:05 -0700 Subject: [PATCH 02/12] Fiona's edits to Distributed Tracing --- .../performance/distributed-tracing.md | 100 +++++++++--------- 1 file changed, 52 insertions(+), 48 deletions(-) diff --git a/src/collections/_documentation/performance/distributed-tracing.md b/src/collections/_documentation/performance/distributed-tracing.md index 62598c0330f27..fe68432a96cc7 100644 --- a/src/collections/_documentation/performance/distributed-tracing.md +++ b/src/collections/_documentation/performance/distributed-tracing.md @@ -12,15 +12,15 @@ Sentry's Performance features are currently in beta. For more details about acce level="warning" %} -Enabling tracing allows you to augment your existing error data by capturing interactions between your software systems. This lets Sentry track valuable metrics about your software performance, like throughput and latency, as well as show the impact of errors across multiple systems. Tracing makes Sentry a more complete monitoring solution, allowing you to diagnose problems more easily and measure your application's overall health, by revealing insights such as: +Enabling tracing augments your existing error data by capturing interactions among your software systems. With tracing, Sentry tracks metrics about your software performance, like throughput and latency, as well as displays the impact of errors across multiple systems. In addition, tracing makes Sentry a more complete monitoring solution, helping you both diagnose problems and measure your application's overall health more easily. Tracing in Sentry provides insights such as: -- What occurred for a specific error event, or issue -- What conditions are causing bottlenecks or latency issues in the application -- Which endpoints or operations consume the most time +- What occurred for a specific error event or issue +- The conditions that cause bottlenecks or latency issues in your application +- The endpoints or operations that consume the most time ## What is Tracing? -Applications typically consist of interconnected components (also called services). Let's take as an example a modern web application, composed of the following components, separated by network boundaries: +Applications typically consist of interconnected components, which are also called services. As an example, let's look at a modern web application, composed of the following components, separated by network boundaries: - Front End (Single-Page Application) - Backend (REST API) @@ -28,47 +28,51 @@ Applications typically consist of interconnected components (also called service - Database Server - Cron Job Scheduler -Each of these components may be written in a different language on a different platform. Each one can be instrumented individually with a Sentry SDK (to capture error data or crash reports) but that doesn't tell the full story because each piece is considered separately. Tracing allows you to tie all of that data together; in our example, that means being able to follow a request from the front end to the backend and back, pulling in data from any background tasks or notification jobs that request creates. Not only does this allow you to correlate Sentry error reports, to see how an error in one service may have propagated to another, it also allows you to get much better insights into which services may be having a negative impact on your applications overall performance. +Each of these components may be written in a different language on a different platform. Each can be instrumented individually using the Sentry SDK to capture error data or crash reports, but that instrumentation doesn't provide the full picture as each piece is considered separately. Tracing allows you to tie all of the data together. + +In our example of a modern web application, tracing means being able to follow a request from the front end to the backend and back, pulling in data from any background tasks or notification jobs a request creates. Not only does this allow you to correlate Sentry error reports, to see how an error in one service may have propagated to another, but this also allows you to gain stronger insights into which services may be having a negative impact on your application's overall performance. Before learning how to enable tracing in your application, it helps to understand a few key terms and how they relate to one another. ### Traces, Transactions, and Spans -A **trace** represents the record of the entire operation you want to measure or track - like page load, an instance of a user completing some action in your application, or a cron job in your backend. When that trace includes work done in multiple services, such as those listed above, it's called a **distributed trace**. +A **trace** represents the record of the entire operation you want to measure or track - like page load, an instance of a user completing some action in your application, or a cron job in your backend. When a trace includes work in multiple services, such as those listed above, it's called a **distributed trace**, because the trace is distributed across those services. To recap, a trace is a record of an operation; a distributed trace is a record of an operation that's distributed across services. -Each trace consists of one or more tree-like structures called **transactions**, the nodes of which are called **spans**. In most cases, each transaction represents a single instance of a service being called, and each span within that transaction represents that service performing a single unit of work, be it calling a function within that service or making a call to a different service. +Each trace consists of one or more tree-like structures called **transactions**, the nodes of which are called **spans**. In most cases, each transaction represents a single instance of a service being called, and each span within that transaction represents that service performing a single unit of work, whether calling a function within that service or making a call to a different service. -Since the transaction has a tree structure, top-level spans can themselves be broken down into smaller spans, mirroring the way that one function may call a number of other, smaller functions; this is expressed using the parent-child metaphor, so that every span may be the **parent span** to multiple other spans. Further, since all trees must have a single root, there is always one span representing the transaction itself, of which all other spans are descendants. +Because a transaction has a tree structure, top-level spans can themselves be broken down into smaller spans, mirroring the way that one function may call a number of other, smaller functions; this is expressed using the parent-child metaphor, so that every span may be the **parent span** to multiple other spans. Further, since all trees must have a single root, one span always represents the transaction itself, of which all other spans are descendants. To make all of this more concrete, let's consider our example web app again. -### Example: Slow Page Load +### Example: Investigating Slow Page Load -Suppose your web application is slow to load, and you'd like to know why. A lot has to happen in order for your app to first get to a usable state: multiple requests to your backend, likely some work - including calls to your database or to outside APis - done there before responses are returned, and processing by the browser in order to render all of the returned data into something meaningful to the user. So which part of that process is slowing things down? +Suppose your web application is slow to load, and you'd like to know why. A lot has to happen for your app to first get to a usable state: multiple requests to your backend, likely some work - including calls to your database or to outside APis - completed before responses are returned, and processing by the browser to render all of the returned data into something meaningful to the user. So which part of that process is slowing things down? -`[kmclb: In what follows I'm not worrying yet about list formatting/indenting, what's a header, what's italics masquerading as a header, etc. Open to opinions here, and also just note-to-self-ing that it needs to be thought about eventually.]` +`[kmclb: In what follows I'm not worrying yet about list formatting/indenting, what's a header, what's italics masquerading as a header, etc. Open to opinions here, and also just note-to-self-ing that it needs to be thought about eventually. FPA: maybe a table? First column, the service and second column, the processing that needs to happen?]` -Let's say that in this simplified example, when a user loads the app in their browser, the following happens in each service: +Let's say, in this simplified example, that when a user loads the app in their browser, the following happens in each service: _Browser_ - - 1 request each for HTML, CSS, and JS + - 1 request each for HTML, CSS, and JavaScript - 1 rendering task, which sets off 2 requests for JSON data _Backend_ - - 3 requests which just require serving static files (the HTML, CSS, and JS) - - 2 requests for JSON data, of which - - 1 requires a call to the database, and + - 3 requests to serve static files (the HTML, CSS, and JS) + - 2 requests for JSON data + - 1 requires a call to the database - 1 requires a call to an external API and work to process the results before returning them to the front end _Database Server_ - - 1 request which requires a query to check authentication and a query to get data + - 1 request which requires two queries + - 1 query to check authentication + - 1 query to get data -(The external API is not listed because it's not yours, and you can't see inside of it.) +_Note:_ The external API is not listed because it's external to your system, and you can't see inside of it. -In this example, the entire page-loading process, including all of the above, would be represented by a single **trace**, and that trace would consist of the following **transactions**: +In this example, the entire page-loading process, including all of the above, is represented by a single **trace**. That trace would consist of the following **transactions**: - 1 browser transaction (for page load) - 5 backend transactions (one for each request) @@ -82,7 +86,7 @@ Each transaction would be broken down into **spans** as follows: - 1 span for the rendering task, which itself contains - 2 child spans, one for each JSON request -Let's pause here to make an important point: Some (though not all) of the browser transaction spans we just listed have a direct correspondence to backend transactions we listed earlier. Specifically, each request _span_ in the browser transaction corresponds to a separate request _transaction_ in the backend. In this situation, when a span in one service gives rise to a transaction in a subsequent service, we call the original span a parent span to _both_ the transaction and its root span. +Let's pause here to make an important point: Some, though not all, of the browser transaction spans listed have a direct correspondence to backend transactions listed earlier. Specifically, each request _span_ in the browser transaction corresponds to a separate request _transaction_ in the backend. In this situation, when a span in one service gives rise to a transaction in a subsequent service, we call the original span a parent span to _both_ the transaction and its root span. In our example, every transaction other than the initial browser page-load transaction is the child of a span in another service, which means that every root span other than the browser transaction root has a parent span (albeit in a different service). In a fully-instrumented system (one in which every service has tracing enabled) this pattern will always hold true: _the only parentless span will be the root of the initial transaction, and of the remaining parented spans, every one will live in the same service as its parent, except for the root spans, whose parents will live in a previous service_. This is worth noting because it is the one way in which transactions are not perfect trees - their roots can (and mostly do) have parents. @@ -98,7 +102,7 @@ Now, for the sake of completeness, back to our spans: _Backend Request with API Call Transaction_: 3 spans - 1 root span representing the entire request (child of a browser span) - - 1 span for the API request (unlike with the DB call, _not_ a parent span, since the API isn't yours) + - 1 span for the API request (unlike with the DB call, _not_ a parent span, since the API is external to your system) - 1 span for processing the API data _Database Server Request Transaction_: 3 spans @@ -106,64 +110,64 @@ Now, for the sake of completeness, back to our spans: - 1 span for the authentication query - 1 span for the query retrieving data -To wrap up the example, after instrumenting all of your services, you might discover that - for some reason - of all things it's the auth query in your database server which is making things slow, accounting for more than half of the time it takes for your entire page load process to complete. Tracing can't tell you _why_ that's happening, but at least now you know where to look! +To wrap up the example, after instrumenting all of your services, you might discover that - for some reason - it's the auth query in your database server that is making things slow, accounting for more than half of the time it takes for your entire page load process to complete. Tracing can't tell you _why_ that's happening, but at least now you know where to look! -`[kmclb: this def needs an illustration (or multiple illustrations) - can we maybe just adjust the current one for this example, or should we use one with more specifics? Current diagram:]` +`[kmclb: this def needs an illustration (or multiple illustrations) - can we maybe just adjust the current one for this example, or should we use one with more specifics? Current diagram: FPA: am asking Sami in Creative, who made the original drawing]` [{% asset performance/tracing-diagram.png alt="Diagram illustrating how a trace is composed of multiple transactions." %}]({% asset performance/tracing-diagram.png @path %}) ### Further Examples -Here are a few more examples of potential usecases for tracing, broken down into transactions and spans: +This section addresses a few more examples of tracing, broken down into transactions and spans: -(Note: starred spans represent spans which are the parent of a later transaction (and its root span)) +_Note:_ Starred spans represent spans that are the parent of a later transaction (and its root span). -**Particular User Actions** +**Measuring Time for E-Commerce** -If you application involves ecommerce, you might want to measure the time between a user clicking "Submit Order" and the order confirmation appearing, including tracking the submitting of the charge to the payment processor and the sending of an order confirmation email. Here that entire process would be one trace, and you'd likely have transactions for: +If your application involves e-commerce, you likely want to measure the time between a user clicking "Submit Order" and the order confirmation appearing, including tracking the submitting of the charge to the payment processor and the sending of an order confirmation email. That entire process is one trace, and typically you'd have transactions for: - The browser's full process (example spans: XHR request to backend\*, rendering confirmation screen) - Your backend's processing of that request (example spans: function call to compute total, DB call to store order*, API call to payment processor, queuing of email confirmation\*) - Your database's work updating the customer's order history (example spans: individual SQL queries) - The queued task of sending the email (example spans: function call to populate email template, API call to email-sending service) -**Background Operations** +**Checking Backend Transactions** -If your backend periodically polls for data from an external service, processes it, caches it, and then forwards it to an internal service, each instance of this happening would be a trace, and you'd likely have transactions for: +If your backend periodically polls for data from an external service, processes it, caches it, and then forwards it to an internal service, each instance of this happening is a trace, and you'd typically have transactions for: -- The cron job which completes the entire process (example spans: API call to external service, processing function, call to caching service\*, API call to internal service\*) +- The cron job that completes the entire process (example spans: API call to external service, processing function, call to caching service\*, API call to internal service\*) - The work done in your caching service (example spans: checking cache for existing data, storing new data in cache) - Your internal service's processing of the request (example spans: anything that service might do to handle the request) ### Tracing Data Model -`[kmclb: do we want to format this in a quote-y kind of way?]` +`[kmclb: do we want to format this in a quote-y kind of way? FPA: like a Note or Alert. I think it would be kind of cool.]` "Show me your flowchart and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won't usually need your flowchart; it'll be obvious." -- [Fred Brooks](https://en.wikipedia.org/wiki/Fred_Brooks), The Mythical Man Month (1975) -All of the above theory is very nice, but traces, transactions, and spans are ultimately defined by what's in them, and the relationship among them is defined by how links between them are recorded. +While the theory is interesting, traces, transactions, and spans are ultimately defined by what's in them, and the relationship among them is defined by how links between them are recorded.`[FPA: this sentence is confusing to me. Do we mean. "Traces, transactions, and spans are defined by what's in them. The relationships among the three are defined by the manner in which the links between them are recorded." or something like that?]` **Traces** -Traces are not an entity in and of themselves. Rather, a trace is defined to be the collection of all transactions which share a `trace_id` value. +Traces are not an entity in and of themselves. Rather, a trace is defined as the collection of all transactions that share a `trace_id` value. **Transactions** -Transactions share most of their properties (start and end time, tags, etc) with their root spans, so the same options are available (see below) and setting them in either place is equivalent. Transactions also have two extra properties not included in spans: +Transactions share most of their properties (start and end time, tags, and so forth) with their root spans, so the same options are available (see below) `[FPA: suggest we say "so the same options described are available]` and setting them in either place is equivalent. Transactions also have two additional properties not included in spans: `[kmclb: anything missing from this list?]` -- `transaction_name`: used in various places in the UI to identify the transaction +- `transaction_name`: used in the UI to identify the transaction - `root_span`: pointer to the root of the transactions span tree -`[kmclb: how the heck do you get less-than or greater-than symbols to show up in code snippets?]` +`[kmclb: how the heck do you get less-than or greater-than symbols to show up in code snippets? FPA: I usually just enclose in backticks, but it can be tricky - you can force also with html tags]` Common examples of `transaction_name` values include endpoint paths (like `/store/checkout/` or `api/v2/users/\/`) for backend request transactions, task names (like `data.cleanup.delete_inactive_users`) for cron job transactions, and URLs (like `https://docs.sentry.io/performance/distributed-tracing/`) for page-load transactions. **Spans** -The majority of the data in a transaction lives in the individual spans it contains. Span data includes: +The majority of the data in a transaction resides in the individual spans the transaction contains. Span data includes: `[kmclb: the list below isn't currently ground truth (quite), but the ground is potentially shifting. Will update once API is settled.]` - - `parent_span_id`: to tie the span to its parent span + - `parent_span_id`: ties the span to its parent span - `op`: short code identifying the type of operation the span is measuring - `start_timestamp`: when the span was opened - `end_timestamp`: when the span was closed @@ -172,30 +176,30 @@ The majority of the data in a transaction lives in the individual spans it conta - `tags`: key-value pairs holding additional data about the span (optional) - `data`: arbitrarily-structured additional data about the span (optional) -An example use of the `op` and `description` properties together is `op: sql.query` and `description: SELECT * FROM users WHERE last_active < DATE_SUB(CURRENT_DATE, INTERVAL 1 YEAR)`. The `status` property is often used to indicate the success or failure of the span's operation (or for a response code in the case of HTTP requests). Finally, `tags` and `data` allow you attach further contextual information to the span, such as `function: middleware.auth.is_authenticated` for a function call or `request: {url: ..., headers: ... , body: ...}` for an http request. +An example use of the `op` and `description` properties together is `op: sql.query` and `description: SELECT * FROM users WHERE last_active < DATE_SUB(CURRENT_DATE, INTERVAL 1 YEAR)`. The `status` property is often used to indicate the success or failure of the span's operation, or for a response code in the case of HTTP requests. Finally, `tags` and `data` allow you attach further contextual information to the span, such as `function: middleware.auth.is_authenticated` for a function call or `request: {url: ..., headers: ... , body: ...}` for an HTTP request. -### Good to Know (needs a better name) +### Good to Know (needs a better name) `[FPA: Tracing Relationships]` -A few more important points about traces, transactions, and spans, and they way they relate to one another: +A few more important points about traces, transactions, and spans, and the way they relate to one another: -**Trace Duration**: Because a trace is just a collection of transactions, traces don't have their own start and end times. Instead, a trace is considered to begin when its earliest transaction starts, and ends when its latest transaction ends. This also means you can't explicitly start or end a trace, only the transactions in that trace. +**Trace Duration**: Because a trace is a collection of transactions, traces don't have their own start and end times. Instead, a trace begins when its earliest transaction starts, and ends when its latest transaction ends. As a result, you can't explicitly start or end a trace, you can only start and end the transactions in that trace. -**Async Transactions**: Because of the possibility of asynchronous processes, child transactions may outlive the transactions containing their parent spans, sometimes by many orders of magnitude. For example, if an backend API call sets off a long-running processing task and then immediately returns a response, the backend transaction will finish (and its data will be sent to Sentry) long before the async task transaction does. Asynchronicity also means that the order in which transactions are sent to (and received by) Sentry is in no way dependent on the order in which they were created. (Because of the variability of transmission times, order of receipt for transactions in the same trace is also only somewhat correlated with order of completion.) +**Async Transactions**: Because of the possibility of asynchronous processes, child transactions may outlive the transactions containing their parent spans, sometimes by many orders of magnitude. For example, if a backend API call sets off a long-running processing task, then immediately returns a response, the backend transaction will finish (and its data will be sent to Sentry) long before the async task transaction does. Asynchronicity also means that the order in which transactions are sent to (and received by) Sentry is not dependent on the order in which they were created. (Because of the variability of transmission times, order of receipt for transactions in the same trace is also only somewhat correlated with order of completion.) -**Orphan Transactions**: In theory, in a fully instrumented system, each trace should only contain one transaction and one span (the transaction's root) without a parent, namely the transaction in the originating service. However, in practice, you may not have tracing enabled in every one of your services, or an instrumented service may fail to report a transaction due to network disruption or other unforeseen circumstances. When this happens, you can end up with gaps in your trace hierarchy, and specifically, you may see transactions partway through the trace whose parent spans haven't been recorded as part of any known transactions. Such non-originating, parentless transactions are called **orphan transactions**. +**Orphan Transactions**: In theory, in a fully instrumented system, each trace should contain only one transaction and one span (the transaction's root) without a parent, namely the transaction in the originating service. However, in practice, you may not have tracing enabled in every one of your services, or an instrumented service may fail to report a transaction due to network disruption or other unforeseen circumstances. When this happens, you may have gaps in your trace hierarchy. Specifically, you may see transactions partway through the trace whose parent spans haven't been recorded as part of any known transactions. Such non-originating, parentless transactions are called **orphan transactions**. `[kmclb: depending on how we end up formatting this, may need to make this one paragraph]` -**Clock Skew**: If you are collecting transactions from multiple machines, you will likely encounter **clock skew**, which is the situation in which timestamps in one transaction don't align with timestamps in another. For example, if your backend makes a database call, the backend transaction logically should start before the database transaction does. But if the system time on the two machines (hosting your backend and database, respectively) aren't synced to common standard, it's possible that won't be the case. It's also possible for the ordering to be correct but or the two recorded timeframes nonetheless to not line up in a way which is accurate to what actually happened. +**Clock Skew**: If you are collecting transactions from multiple machines, you will likely encounter **clock skew**, where timestamps in one transaction don't align with timestamps in another. For example, if your backend makes a database call, the backend transaction logically should start before the database transaction does. But if the system time on the two machines (hosting your backend and database, respectively) aren't synced to common standard, it's possible that won't be the case. It's also possible for the ordering to be correct, but for the two recorded timeframes to not line up in a way that accurately reflects what actually happened. Because there is no way for Sentry to judge either the relative or absolute accuracy of your timestamps, it does not attempt to correct or modify them in any way. And while you can reduce clock skew by using Network Time Protocol (NTP) or your cloud provider's clock synchronization services, you may still notice small drifts in your data, as synchronizing clocks on small intervals is challenging. -**Nesting Spans**: Though our examples above had four levels in their hierarchy (trace, transaction, span, child span) there's no set limit to how deep the nesting of spans can go. There are, however, practical limits: transaction payloads sent to Sentry have a maximum allowed size (currently `[kmclb: find out what this is]`), and there's a balance to be struck between your data's granularity and its usability. +**Nesting Spans**: Though our examples above had four levels in their hierarchy (trace, transaction, span, child span) there's no set limit to how deep the nesting of spans can go. There are, however, practical limits: transaction payloads sent to Sentry have a maximum allowed size (currently `[kmclb: find out what this is]`), as there's a balance to be struck between your data's granularity and its usability. **Zero-duration Spans**: It's possible for a span to have equal start and end times, and therefore be recorded as taking no time. This can occur either because the span is being used as a marker (such as is done in [the browser's Performance API](https://developer.mozilla.org/en-US/docs/Web/API/Performance/mark)) or because the amount of time the operation took is less than the measurement resolution (which will vary by service). **What We Send**: Individual spans aren't sent to Sentry. Rather, those spans are attached to their containing transaction, and the transaction is sent as one unit. This means that no span data will be recorded by Sentry's servers until the transaction to which they belong is closed and dispatched. -`[kmclb: STOP READING AT THIS POINT. Or, keep reding if you want, but I haven't worked on anything below here in a concerted way yet, so for the purposes of this initial feedback session, I'm really just looking for comments on ^^^^^^^^^]` +`[kmclb: STOP READING AT THIS POINT. Or, keep reding if you want, but I haven't worked on anything below here in a concerted way yet, so for the purposes of this initial feedback session, I'm really just looking for comments on ^^^^^^^^^] FPA: got it :)` ## Data Sampling From dcb2e918250b0595ca8c61016ecd6317aa832218 Mon Sep 17 00:00:00 2001 From: Katie Byers Date: Mon, 4 May 2020 12:35:41 -0700 Subject: [PATCH 03/12] katie's edits of fiona's edits --- .../performance/distributed-tracing.md | 34 +++++++++---------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/src/collections/_documentation/performance/distributed-tracing.md b/src/collections/_documentation/performance/distributed-tracing.md index fe68432a96cc7..4d5ada81b7842 100644 --- a/src/collections/_documentation/performance/distributed-tracing.md +++ b/src/collections/_documentation/performance/distributed-tracing.md @@ -28,9 +28,9 @@ Applications typically consist of interconnected components, which are also call - Database Server - Cron Job Scheduler -Each of these components may be written in a different language on a different platform. Each can be instrumented individually using the Sentry SDK to capture error data or crash reports, but that instrumentation doesn't provide the full picture as each piece is considered separately. Tracing allows you to tie all of the data together. +Each of these components may be written in a different language on a different platform. Each can be instrumented individually using a Sentry SDK to capture error data or crash reports, but that instrumentation doesn't provide the full picture, as each piece is considered separately. Tracing allows you to tie all of the data together. -In our example of a modern web application, tracing means being able to follow a request from the front end to the backend and back, pulling in data from any background tasks or notification jobs a request creates. Not only does this allow you to correlate Sentry error reports, to see how an error in one service may have propagated to another, but this also allows you to gain stronger insights into which services may be having a negative impact on your application's overall performance. +In our example web application, tracing means being able to follow a request from the front end to the backend and back, pulling in data from any background tasks or notification jobs that request creates. Not only does this allow you to correlate Sentry error reports, to see how an error in one service may have propagated to another, but it also allows you to gain stronger insights into which services may be having a negative impact on your application's overall performance. Before learning how to enable tracing in your application, it helps to understand a few key terms and how they relate to one another. @@ -66,11 +66,11 @@ Let's say, in this simplified example, that when a user loads the app in their b _Database Server_ - - 1 request which requires two queries + - 1 request which requires 2 queries - 1 query to check authentication - 1 query to get data -_Note:_ The external API is not listed because it's external to your system, and you can't see inside of it. +_Note:_ The external API is not listed precisely because it's external, and you therefore can't see inside of it. In this example, the entire page-loading process, including all of the above, is represented by a single **trace**. That trace would consist of the following **transactions**: @@ -102,7 +102,7 @@ Now, for the sake of completeness, back to our spans: _Backend Request with API Call Transaction_: 3 spans - 1 root span representing the entire request (child of a browser span) - - 1 span for the API request (unlike with the DB call, _not_ a parent span, since the API is external to your system) + - 1 span for the API request (unlike with the DB call, _not_ a parent span, since the API is external) - 1 span for processing the API data _Database Server Request Transaction_: 3 spans @@ -117,11 +117,11 @@ To wrap up the example, after instrumenting all of your services, you might disc ### Further Examples -This section addresses a few more examples of tracing, broken down into transactions and spans: +This section contains a few more examples of tracing, broken down into transactions and spans. _Note:_ Starred spans represent spans that are the parent of a later transaction (and its root span). -**Measuring Time for E-Commerce** +**Measuring a Specific User Action** If your application involves e-commerce, you likely want to measure the time between a user clicking "Submit Order" and the order confirmation appearing, including tracking the submitting of the charge to the payment processor and the sending of an order confirmation email. That entire process is one trace, and typically you'd have transactions for: @@ -130,7 +130,7 @@ If your application involves e-commerce, you likely want to measure the time bet - Your database's work updating the customer's order history (example spans: individual SQL queries) - The queued task of sending the email (example spans: function call to populate email template, API call to email-sending service) -**Checking Backend Transactions** +**Monitoring a Background Process** If your backend periodically polls for data from an external service, processes it, caches it, and then forwards it to an internal service, each instance of this happening is a trace, and you'd typically have transactions for: @@ -143,7 +143,7 @@ If your backend periodically polls for data from an external service, processes `[kmclb: do we want to format this in a quote-y kind of way? FPA: like a Note or Alert. I think it would be kind of cool.]` "Show me your flowchart and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won't usually need your flowchart; it'll be obvious." -- [Fred Brooks](https://en.wikipedia.org/wiki/Fred_Brooks), The Mythical Man Month (1975) -While the theory is interesting, traces, transactions, and spans are ultimately defined by what's in them, and the relationship among them is defined by how links between them are recorded.`[FPA: this sentence is confusing to me. Do we mean. "Traces, transactions, and spans are defined by what's in them. The relationships among the three are defined by the manner in which the links between them are recorded." or something like that?]` +While the theory is interesting, ultimately any data structure is defined by the kind of data it contains, and relationships between data structures are defined by how links between them are recorded. Traces, transactions, and spans are no different. `[FPA: this sentence is confusing to me. Do we mean. "Traces, transactions, and spans are defined by what's in them. The relationships among the three are defined by the manner in which the links between them are recorded." or something like that? kmclb: Does this read more easily to you?]` **Traces** @@ -151,14 +151,14 @@ Traces are not an entity in and of themselves. Rather, a trace is defined as the **Transactions** -Transactions share most of their properties (start and end time, tags, and so forth) with their root spans, so the same options are available (see below) `[FPA: suggest we say "so the same options described are available]` and setting them in either place is equivalent. Transactions also have two additional properties not included in spans: +Transactions share most of their properties (start and end time, tags, and so forth) with their root spans, so the same options described below for spans are available in transactions, and setting them in either place is equivalent. `[FPA: suggest we say "so the same options described are available kmclb: A link felt like overkill for something which is 5 lines below, but hopefully the new wording makes it crystal clear which options we're talking about]` Transactions also have two additional properties not included in spans: `[kmclb: anything missing from this list?]` - `transaction_name`: used in the UI to identify the transaction - `root_span`: pointer to the root of the transactions span tree -`[kmclb: how the heck do you get less-than or greater-than symbols to show up in code snippets? FPA: I usually just enclose in backticks, but it can be tricky - you can force also with html tags]` +`[kmclb: how the heck do you get less-than or greater-than symbols to show up in code snippets? FPA: I usually just enclose in backticks, but it can be tricky - you can force also with html tags kmclb: how do you mean, enclose in backticks? How would you handle the user_id example below?]` Common examples of `transaction_name` values include endpoint paths (like `/store/checkout/` or `api/v2/users/\/`) for backend request transactions, task names (like `data.cleanup.delete_inactive_users`) for cron job transactions, and URLs (like `https://docs.sentry.io/performance/distributed-tracing/`) for page-load transactions. **Spans** @@ -178,22 +178,22 @@ The majority of the data in a transaction resides in the individual spans the tr An example use of the `op` and `description` properties together is `op: sql.query` and `description: SELECT * FROM users WHERE last_active < DATE_SUB(CURRENT_DATE, INTERVAL 1 YEAR)`. The `status` property is often used to indicate the success or failure of the span's operation, or for a response code in the case of HTTP requests. Finally, `tags` and `data` allow you attach further contextual information to the span, such as `function: middleware.auth.is_authenticated` for a function call or `request: {url: ..., headers: ... , body: ...}` for an HTTP request. -### Good to Know (needs a better name) `[FPA: Tracing Relationships]` +### Good to Know (needs a better name) `[FPA: Tracing Relationships kmclb: but that's really what this entire (big) section is about. This smaller section is more "fun factoids which are helpful but didn't fit anywhere else" - some are about relationships between traces, transactions, and spans, but some are just about one of those]` A few more important points about traces, transactions, and spans, and the way they relate to one another: -**Trace Duration**: Because a trace is a collection of transactions, traces don't have their own start and end times. Instead, a trace begins when its earliest transaction starts, and ends when its latest transaction ends. As a result, you can't explicitly start or end a trace, you can only start and end the transactions in that trace. +**Trace Duration**: Because a trace just is a collection of transactions, traces don't have their own start and end times. Instead, a trace begins when its earliest transaction starts, and ends when its latest transaction ends. As a result, you can't explicitly start or end a trace directly. Instead, you create a trace by creating the first transaction in that trace, and you complete a trace by completing all of transactions it contains. -**Async Transactions**: Because of the possibility of asynchronous processes, child transactions may outlive the transactions containing their parent spans, sometimes by many orders of magnitude. For example, if a backend API call sets off a long-running processing task, then immediately returns a response, the backend transaction will finish (and its data will be sent to Sentry) long before the async task transaction does. Asynchronicity also means that the order in which transactions are sent to (and received by) Sentry is not dependent on the order in which they were created. (Because of the variability of transmission times, order of receipt for transactions in the same trace is also only somewhat correlated with order of completion.) +**Async Transactions**: Because of the possibility of asynchronous processes, child transactions may outlive the transactions containing their parent spans, sometimes by many orders of magnitude. For example, if a backend API call sets off a long-running processing task and then immediately returns a response, the backend transaction will finish (and its data will be sent to Sentry) long before the async task transaction does. Asynchronicity also means that the order in which transactions are sent to (and received by) Sentry does not in any way depend on the order in which they were created. (By contrast, order of receipt for transactions in the same trace _is_ correlated with order of completion, though because of factors like the variability of transmission times, the correlation is far from perfect.) -**Orphan Transactions**: In theory, in a fully instrumented system, each trace should contain only one transaction and one span (the transaction's root) without a parent, namely the transaction in the originating service. However, in practice, you may not have tracing enabled in every one of your services, or an instrumented service may fail to report a transaction due to network disruption or other unforeseen circumstances. When this happens, you may have gaps in your trace hierarchy. Specifically, you may see transactions partway through the trace whose parent spans haven't been recorded as part of any known transactions. Such non-originating, parentless transactions are called **orphan transactions**. +**Orphan Transactions**: In theory, in a fully instrumented system, each trace should contain only one transaction and one span (the transaction's root) without a parent, namely the transaction in the originating service. However, in practice, you may not have tracing enabled in every one of your services, or an instrumented service may fail to report a transaction due to network disruption or other unforeseen circumstances. When this happens, you may see gaps in your trace hierarchy. Specifically, you may see transactions partway through the trace whose parent spans haven't been recorded as part of any known transactions. Such non-originating, parentless transactions are called **orphan transactions**. `[kmclb: depending on how we end up formatting this, may need to make this one paragraph]` -**Clock Skew**: If you are collecting transactions from multiple machines, you will likely encounter **clock skew**, where timestamps in one transaction don't align with timestamps in another. For example, if your backend makes a database call, the backend transaction logically should start before the database transaction does. But if the system time on the two machines (hosting your backend and database, respectively) aren't synced to common standard, it's possible that won't be the case. It's also possible for the ordering to be correct, but for the two recorded timeframes to not line up in a way that accurately reflects what actually happened. +**Clock Skew**: If you are collecting transactions from multiple machines, you will likely encounter **clock skew**, where timestamps in one transaction don't align with timestamps in another. For example, if your backend makes a database call, the backend transaction logically should start before the database transaction does. But if the system time on each machine (those hosting your backend and database, respectively) isn't synced to common standard, it's possible that won't be the case. It's also possible for the ordering to be correct, but for the two recorded timeframes to not line up in a way that accurately reflects what actually happened. Because there is no way for Sentry to judge either the relative or absolute accuracy of your timestamps, it does not attempt to correct or modify them in any way. And while you can reduce clock skew by using Network Time Protocol (NTP) or your cloud provider's clock synchronization services, you may still notice small drifts in your data, as synchronizing clocks on small intervals is challenging. -**Nesting Spans**: Though our examples above had four levels in their hierarchy (trace, transaction, span, child span) there's no set limit to how deep the nesting of spans can go. There are, however, practical limits: transaction payloads sent to Sentry have a maximum allowed size (currently `[kmclb: find out what this is]`), as there's a balance to be struck between your data's granularity and its usability. +**Nesting Spans**: Though our examples above had four levels in their hierarchy (trace, transaction, span, child span) there's no set limit to how deep the nesting of spans can go. There are, however, practical limits: transaction payloads sent to Sentry have a maximum allowed size (currently `[kmclb: find out what this is]`), and there's a balance to be struck between your data's granularity and its usability. **Zero-duration Spans**: It's possible for a span to have equal start and end times, and therefore be recorded as taking no time. This can occur either because the span is being used as a marker (such as is done in [the browser's Performance API](https://developer.mozilla.org/en-US/docs/Web/API/Performance/mark)) or because the amount of time the operation took is less than the measurement resolution (which will vary by service). From d5355f33946adaf9094d1b358f1587ca22eae82e Mon Sep 17 00:00:00 2001 From: Katie Byers Date: Mon, 4 May 2020 20:21:09 -0700 Subject: [PATCH 04/12] add profiling v tracing, take a pass through middle section --- .../performance/distributed-tracing.md | 77 +++++++++++++------ 1 file changed, 53 insertions(+), 24 deletions(-) diff --git a/src/collections/_documentation/performance/distributed-tracing.md b/src/collections/_documentation/performance/distributed-tracing.md index 4d5ada81b7842..bbb0ffcf9cbc9 100644 --- a/src/collections/_documentation/performance/distributed-tracing.md +++ b/src/collections/_documentation/performance/distributed-tracing.md @@ -12,7 +12,7 @@ Sentry's Performance features are currently in beta. For more details about acce level="warning" %} -Enabling tracing augments your existing error data by capturing interactions among your software systems. With tracing, Sentry tracks metrics about your software performance, like throughput and latency, as well as displays the impact of errors across multiple systems. In addition, tracing makes Sentry a more complete monitoring solution, helping you both diagnose problems and measure your application's overall health more easily. Tracing in Sentry provides insights such as: +Enabling tracing augments your existing error data by capturing interactions among your software systems. With tracing, Sentry tracks metrics about your software performance, like throughput and latency, as well as displays the impact of errors across multiple systems. Tracing makes Sentry a more complete monitoring solution, helping you both diagnose problems and measure your application's overall health more easily. Tracing in Sentry provides insights such as: - What occurred for a specific error event or issue - The conditions that cause bottlenecks or latency issues in your application @@ -20,6 +20,14 @@ Enabling tracing augments your existing error data by capturing interactions amo ## What is Tracing? +To begin, a note about what tracing is not: Tracing is not profiling. Though the goals of profiling and tracing overlap quite a bit, and though they are both tools which can be used to diagnose problems in your application, they differ both in terms of what they measure and how the data is recorded. + +A [profiler](https://en.wikipedia.org/wiki/Profiling_(computer_programming)) may measure any number of aspects of an application's operation: the number of instructions executed, the amount of memory being used by various processes, the amount of time a given function call takes, and many more. The resulting profile is a statistical summary of these measurements. + +A [tracing tool](https://en.wikipedia.org/wiki/Tracing_(software)), on the other hand, focuses on _what_ happened (and when), rather than how many times it happened or how long it took. The resulting trace is log of events which occurred during a program's execution, often across multiple systems. Though traces most often - or, in the case of Sentry's traces, always - include timestamps, allowing durations to be calculated, measuring performance is not their only purpose. They can also show the ways in which interconnected systems interact, and the ways in which problems in one can cause problems in another. + +### Why Tracing? + Applications typically consist of interconnected components, which are also called services. As an example, let's look at a modern web application, composed of the following components, separated by network boundaries: - Front End (Single-Page Application) @@ -199,54 +207,72 @@ Because there is no way for Sentry to judge either the relative or absolute accu **What We Send**: Individual spans aren't sent to Sentry. Rather, those spans are attached to their containing transaction, and the transaction is sent as one unit. This means that no span data will be recorded by Sentry's servers until the transaction to which they belong is closed and dispatched. -`[kmclb: STOP READING AT THIS POINT. Or, keep reding if you want, but I haven't worked on anything below here in a concerted way yet, so for the purposes of this initial feedback session, I'm really just looking for comments on ^^^^^^^^^] FPA: got it :)` - ## Data Sampling -**When collecting traces, we strongly recommend sampling your data.** +When you enable sampling in your tracing setup, you choose a percentage of collected transactions to send to Sentry. For example, if you had an endpoint that received 1000 requests per minute, a sampling rate of 0.25 would result in approximately 250 transactions (25%) being sent to Sentry. (The number is approximate because each request is either tracked or not, independently and pseudorandomly, with a 25% probability. So in the same way that 100 fair coins, when flipped, will result in approximately 50 heads, the SDK will decide to collect a trace in approximately 250 cases.) Because you know the sampling percentage, you can then extrapolate your total traffic volume. + +When collecting traces, we **strongly recommend** sampling your data, for two reasons. First, though capturing a single trace involves minimal overhead, capturing traces for every single pageload, or every single API request, has the potential to add an undesirable amount of load to your system. Second, by enabling sampling you'll more easily prevent yourself from exceeding your organization's [event quota]({%- link _documentation/accounts/quotas/index.md -%}), which will help you manage costs. + +When choosing a sampling rate, the goal is to not collect _too_ much data (given the reasons above) but also to collect enough data that you are able to draw meaningful conclusions. If you're not sure what rate to choose, we recommend starting with a low value and gradually increasing it as you learn more about your traffic patterns and volume, until you've found a rate which lets you balance performance and cost concerns with data accuracy. + +### Consistency Across Transactions Within a Trace + +For traces which involve multiple transactions, Sentry uses a "head-based" approach: a sampling decision is made in the originating service, and then that decision is passed to all subsequent services. To see how this works, let's return to our webapp example above. Consider two users, A and B, who are both loading your app in their respective browsers. When A loads the app, the SDK pseudorandomly "decides" to collect a trace, whereas when B loads the app, the SDK "decides" not to. When each browser makes requests to your backend, it includes in those requests the "yes, please collect transactions" or the "no, not collecting transactions this time" decision in the headers. -When you enable sampling for transaction events in Sentry, you choose a percentage of collected transactions to send to Sentry. For example, if you had an endpoint that received 1000 requests per minute, a sampling rate of 0.25 would result in 250 transactions (25%) being sent to Sentry. +When your backend processes the requests from A's browser, it will see the "yes" decision, collect transaction and span data, and send it to Sentry. Further, it will include the "yes" decision in any requests it makes to subsequent services (like your database server), which will similarly collect data, send it to Sentry, and pass the decision along to any services they call. Though this process, all of the relevant transactions in A's trace will be collected and sent to Sentry. -Sampling enables you to collect traces on a subset of your traffic and extrapolate to the total volume. Furthermore, **enabling sampling allows you to control the volume of data you send to Sentry and lets you better manage your costs**. If you don't have a good understanding of what sampling rate to choose, we recommend you start with a low value and gradually increase the sampling rate as you learn more about your traffic patterns and volume. +On the other hand, when your backend processes the requests from B's browser, it will see the "no" decision, and as a result it will _not_ collect and send transaction and span data to Sentry. It will, however, do the same thing it did in A's case in terms of propagating the decision to subsequent services, telling them not to collect or send data either. They will in turn tell any services they call not to send data, and in this way no transactions from B's trace will be collected. -When you have multiple projects collecting transaction events, Sentry utilizes "head-based" sampling to ensure that once a sampling decision has been made at the beginning of the trace (typically the initial transaction), that decision is propagated to each service or project involved in the trace. If your services have multiple entry points, you should aim to choose a consistent sampling rate for each, as choosing different sampling rates can bias your results. Sentry does not support "tail-based" sampling at this time. +Put simply: as a result of this head-based approach, where the decision is made once in the originating service and passed to all subsequent services, either all of the transactions for a given trace will be collected, or none will, so there shouldn't be any incomplete traces. -If you enable Performance collection for a large portion of your traffic, you may exceed your organization's [Quotas and Rate Limits]({%- link _documentation/accounts/quotas/index.md -%}). +### Consistency Across Services + +If you enable tracing in services with multiple entry points, we recommend choosing similar sampling rates, to avoid biasing your data. For example, suppose the backend of our on-going webapp example also serves as a public API. In that case, some traces would start with a pageload transaction in the web app, and likely include multiple backend transactions, while other traces (those representing folks hitting the public API) would begin with a single backend request transaction, which would be the only backend transaction in the trace. Choosing a very different sampling rate for your webapp from that chosen for your backend would lead to one of those scenarios being oversampled compared to the other, compromising the accuracy of your overall data. ## Viewing Trace Data You can see a list of transaction events by clicking on the "Transactions" pre-built query in [Discover]({%- link _documentation/performance/discover/index.md -%}), or by using a search condition `event.type:transaction` in the [Discover Query Builder]({%- link _documentation/performance/discover/query-builder.md -%}) view. -`[kmclb: I think we should have a section here on searching, which would mention that transactions (but not spans) are searchable, and what data one can use to search. Should also probably link to the main search docs.]` +### Transaction List View + +`[kmclb: There's a LOT we don't document currently about the transaction list view (which is to say, almost anything). Some of it may be covered in the Discover docs, but even if so, I think we need to do a quick overview of all of the controls/what's displayed and then point to those docs. ]` + +`[kmclb: I think we should have a section - or at least a sentence or two - here on searching, which would mention that transactions (but not spans) are searchable, and what data one can use to search. Should also probably link to the main search docs.]` -### Performance Metrics +`[kmclb: We probably also want to explain how transactions are grouped in this view]` -`[kmclb: we don't actually tell you *how* to do this...]` -Some aggregate functions can be applied to transaction events to help you better understand the performance characteristics of your applications. +#### Performance Metrics -**Duration Percentiles** +`[kmclb: Help! This is a dangling modifier (there's a "you" implied by the first clause which never materializes in the second) but I can't think of a good way to fix it.]` +When choosing which columns to display in your transaction list, there are a few metrics which lend themselves well to analyzing your application's performance. +`[kmclb: Do we want to mention the count() option? see note above above grouping]` + +**Aggregate Duration Metrics** You can aggregate transaction durations using the following functions: - average -- 75%, 95%, and 99% duration percentiles +- various percentiles (by default, the pre-built Transactions query shows the 75th and 95th percentiles, but there are many other options, including displaying a custom percentile) One use case for tracking these statistics is to help you identify transactions that are slower than your organization's target SLAs. -A word of caution when looking at averages and percentiles: In most cases, you'll want to set up tracing so that only a fraction of possible traces are actually sent to Sentry, to avoid overwhelming your system. `[kmclb: include a link to sampling section]` Further, you may want to filter your transaction data by date or other factors, or you may be tracing a relatively uncommon operation. For all of these reasons, you may end up with average and percentile data that is directionally correct, but not accurate. (To use the most extreme case as an example, if only a single transaction matches your filters, you can still compute an "average" duration, even though that's clearly not what is usually meant by "average.") +A word of caution when looking at averages and percentiles: In most cases, you'll want to set up tracing so that only [a fraction](#data-sampling) of possible traces are actually sent to Sentry, to avoid overwhelming your system. Further, you may want to filter your transaction data by date or other factors, or you may be tracing a relatively uncommon operation. For all of these reasons, you may end up with average and percentile data that is directionally correct, but not accurate. (To use the most extreme case as an example, if only a single transaction matches your filters, you can still compute an "average" duration, even though that's clearly not what is usually meant by "average.") -`[kmclb: "We can calculate an average with less data than a 95th percentile" - is there some threshold below which we won't do the calculation?]` +`[kmclb: Below: "We can calculate an average with less data than a 95th percentile" - is there some threshold below which we won't do the calculation?]` -The problem of small sample size (and the resulting inability to be usefully accurate) will happen more often in some columns than others. `[kmclb: which ones?]` We can calculate an average with less data than it takes to calculate the 95th percentile, but it’ll also vary by row. (For example, `/settings/` will always get more traffic than `/settings/country/belgium/tax`.) +The problem of small sample size (and the resulting inability to be usefully accurate) will happen more often for some metrics than others. `[kmclb: which ones?]` We can calculate an average with less data than it takes to calculate the 95th percentile, but it’ll also vary by row. (For example, `/settings/` will always get more traffic than `/settings/country/belgium/tax`.) **Requests Per Minute (RPM)** +`[kmclb: is there a reason we talk about RPM but not RPS?]` + Requests Per Minute is a way to measure throughput. It is the average of all request durations, bucketed by the minute `[kmclb: start time or end time?]` for the current time window and query string. +`[kmclb: now that I read this again, it doesn't make sense. Requests/min is a *number* of requests, whereas the description says it's the average *duration* of a group of requests. Which is it?]` ### Transaction Detail View -When you open a transaction event in Discover (by clicking on the ), you'll see the **span view** at the top of the page. Other information the SDK captured as part of the transaction event, such as the transaction's tags and automatically collected breadcrumbs, is displayed underneath and to the right of the span view. +When you open a transaction event in Discover (by clicking on the icon at the left side of the row), you'll see the **span view** at the top of the page. Other information the SDK captured as part of the transaction event (such as the transaction's tags and automatically collected breadcrumbs) is displayed underneath and to the right of the span view. [{% asset performance/discover-span.png alt="Discover span showing the map of the transactions (aka minimap) and the black dashed handlebars for adjusting the window selection." %}]({% asset performance/discover-span.png @path %}) @@ -262,26 +288,29 @@ As shown in the Discover span screenshot above, you can click and drag your mous **Missing Instrumentation** -Sentry may indicate that gaps between spans are "Missing Instrumentation." This message could mean that the SDK was unable to capture or instrument any spans within this gap automatically. It may require you to instrument the gap [manually](#setting-up-tracing). +Sentry may indicate that gaps between spans are "Missing Instrumentation." This means that there is time in the transaction which isn't accounted for by any of the transaction's spans, and likely means that you need to manually instrument that part of your process. **Viewing Span Details** -Click on a row to expand the details of the span. From here, you can see all attached properties, such as **tags** and **data**. +Click on a row in the span view to expand the details of that span. From here, you can see all attached properties, such as tags and data. [{% asset performance/span-detail-view.png alt="Span detail view shows the span id, trace id, parent span id, and other data such as tags." %}]({% asset performance/span-detail-view.png @path %}) **Search by Trace ID** -You can search using `trace id` by expanding any of the span details and clicking on "Search by Trace". +You can search for all of the transactions in a given trace by expanding any of the span details and clicking on "Search by Trace". -You need **project permissions for each project** to see all the transactions within a trace. Each transaction in a trace is likely a part of a different project. If you don't have project permissions, some transactions may not display as part of a trace. +_Note:_ Each transaction belongs to a specific project, and you will only be able to see transactions belonging to projects you have permission to view, which may or may not be all of the given trace's transactions. **Traversing to Child Transactions** -Child transactions are shown based on your project permissions -- which are necessary to viewing transaction events. To check project permissions, navigate to **Settings >> [your organization] >> [your project] >> General**. - Some spans within a transaction may be the parent of another transaction. When you expand the span details for such spans, you'll see the "View Child" button, which, when clicked, will lead to the child transaction's details view. +_Note:_ Each transaction belongs to a specific project, and you will only be able to see transactions belonging to projects you have permission to view, which may or may not include a given span's child transaction. `[kmclb: Does the button even show up if you don't have the right permissions?]` + + +`[kmclb: Haven't really touched much below this. It needs real work, but that's the next PR. :-) ]` + ## Setting Up Tracing {% capture __alert_content -%} From 6341f6927f52237d2d1f575e407817577424ea80 Mon Sep 17 00:00:00 2001 From: Katie Byers Date: Thu, 7 May 2020 08:26:33 -0700 Subject: [PATCH 05/12] final major changes (hopefully) --- .../data-management/advanced-datascrubbing.md | 2 +- .../performance/distributed-tracing.md | 207 ++++++++++-------- 2 files changed, 113 insertions(+), 96 deletions(-) diff --git a/src/collections/_documentation/data-management/advanced-datascrubbing.md b/src/collections/_documentation/data-management/advanced-datascrubbing.md index c500c8ece6f54..fa3387df7e847 100644 --- a/src/collections/_documentation/data-management/advanced-datascrubbing.md +++ b/src/collections/_documentation/data-management/advanced-datascrubbing.md @@ -160,7 +160,7 @@ Select known parts of the schema using the following: * `$logentry` matches the `logentry` attribute of an event. * `$thread` matches a single thread instance in `{"threads": {"values": [...]}}` * `$breadcrumb` matches a single breadcrumb in `{"breadcrumbs": {"values": [...]}}` -* `$span` matches a [trace span]({% link _documentation/performance/performance-glossary.md %}#span) +* `$span` matches a [trace span]({% link _documentation/performance/distributed-tracing.md %}#traces-transactions-and-spans) * `$sdk` matches the SDK context in `{"sdk": ...}` ### Escaping Special Characters diff --git a/src/collections/_documentation/performance/distributed-tracing.md b/src/collections/_documentation/performance/distributed-tracing.md index bbb0ffcf9cbc9..f9f0e42ff22c6 100644 --- a/src/collections/_documentation/performance/distributed-tracing.md +++ b/src/collections/_documentation/performance/distributed-tracing.md @@ -48,7 +48,7 @@ A **trace** represents the record of the entire operation you want to measure or Each trace consists of one or more tree-like structures called **transactions**, the nodes of which are called **spans**. In most cases, each transaction represents a single instance of a service being called, and each span within that transaction represents that service performing a single unit of work, whether calling a function within that service or making a call to a different service. -Because a transaction has a tree structure, top-level spans can themselves be broken down into smaller spans, mirroring the way that one function may call a number of other, smaller functions; this is expressed using the parent-child metaphor, so that every span may be the **parent span** to multiple other spans. Further, since all trees must have a single root, one span always represents the transaction itself, of which all other spans are descendants. +Because a transaction has a tree structure, top-level spans can themselves be broken down into smaller spans, mirroring the way that one function may call a number of other, smaller functions; this is expressed using the parent-child metaphor, so that every span may be the **parent span** to multiple other spans. Further, since all trees must have a single root, one span always represents the transaction itself, and all other spans are descendants of that root span. To make all of this more concrete, let's consider our example web app again. @@ -56,24 +56,19 @@ To make all of this more concrete, let's consider our example web app again. Suppose your web application is slow to load, and you'd like to know why. A lot has to happen for your app to first get to a usable state: multiple requests to your backend, likely some work - including calls to your database or to outside APis - completed before responses are returned, and processing by the browser to render all of the returned data into something meaningful to the user. So which part of that process is slowing things down? -`[kmclb: In what follows I'm not worrying yet about list formatting/indenting, what's a header, what's italics masquerading as a header, etc. Open to opinions here, and also just note-to-self-ing that it needs to be thought about eventually. FPA: maybe a table? First column, the service and second column, the processing that needs to happen?]` - Let's say, in this simplified example, that when a user loads the app in their browser, the following happens in each service: - _Browser_ - +- _Browser_ - 1 request each for HTML, CSS, and JavaScript - 1 rendering task, which sets off 2 requests for JSON data - - _Backend_ - +^ +- _Backend_ - 3 requests to serve static files (the HTML, CSS, and JS) - 2 requests for JSON data - - 1 requires a call to the database - - 1 requires a call to an external API and work to process the results before returning them to the front end - - _Database Server_ - + - 1 requiring a call to the database + - 1 requiring a call to an external API and work to process the results before returning them to the front end +^ +- _Database Server_ - 1 request which requires 2 queries - 1 query to check authentication - 1 query to get data @@ -82,74 +77,97 @@ _Note:_ The external API is not listed precisely because it's external, and you In this example, the entire page-loading process, including all of the above, is represented by a single **trace**. That trace would consist of the following **transactions**: - - 1 browser transaction (for page load) - - 5 backend transactions (one for each request) - - 1 database server transaction (for the single DB request) +- 1 browser transaction (for page load) +- 5 backend transactions (one for each request) +- 1 database server transaction (for the single DB request) Each transaction would be broken down into **spans** as follows: - _Browser Page-load Transaction_: 7 spans +- _Browser Page-load Transaction_: 7 spans - 1 root span representing the entire page load - 1 span each (3 total) for the HTML, CSS, and JS requests - 1 span for the rendering task, which itself contains - 2 child spans, one for each JSON request - + Let's pause here to make an important point: Some, though not all, of the browser transaction spans listed have a direct correspondence to backend transactions listed earlier. Specifically, each request _span_ in the browser transaction corresponds to a separate request _transaction_ in the backend. In this situation, when a span in one service gives rise to a transaction in a subsequent service, we call the original span a parent span to _both_ the transaction and its root span. In our example, every transaction other than the initial browser page-load transaction is the child of a span in another service, which means that every root span other than the browser transaction root has a parent span (albeit in a different service). In a fully-instrumented system (one in which every service has tracing enabled) this pattern will always hold true: _the only parentless span will be the root of the initial transaction, and of the remaining parented spans, every one will live in the same service as its parent, except for the root spans, whose parents will live in a previous service_. This is worth noting because it is the one way in which transactions are not perfect trees - their roots can (and mostly do) have parents. Now, for the sake of completeness, back to our spans: - _Backend HTML/CSS/JS Request Transactions_: 2 spans each `[kmclb: if transactions can have no children, this might be only 1 span, the root]` +- _Backend HTML/CSS/JS Request Transactions_: 1 span each - 1 root span representing the entire request (child of a browser span) - - 1 span for serving the static file - - _Backend Request with DB Call Transaction_: 2 spans +^ +- _Backend Request with DB Call Transaction_: 2 spans - 1 root span representing the entire request (child of a browser span) - 1 span for querying the database (parent of the database server transaction) - - _Backend Request with API Call Transaction_: 3 spans +^ +- _Backend Request with API Call Transaction_: 3 spans - 1 root span representing the entire request (child of a browser span) - 1 span for the API request (unlike with the DB call, _not_ a parent span, since the API is external) - 1 span for processing the API data - - _Database Server Request Transaction_: 3 spans +^ +- _Database Server Request Transaction_: 3 spans - 1 root span representing the entire request (child of the backend span above) - 1 span for the authentication query - 1 span for the query retrieving data - -To wrap up the example, after instrumenting all of your services, you might discover that - for some reason - it's the auth query in your database server that is making things slow, accounting for more than half of the time it takes for your entire page load process to complete. Tracing can't tell you _why_ that's happening, but at least now you know where to look! -`[kmclb: this def needs an illustration (or multiple illustrations) - can we maybe just adjust the current one for this example, or should we use one with more specifics? Current diagram: FPA: am asking Sami in Creative, who made the original drawing]` +To wrap up the example: after instrumenting all of your services, you might discover that - for some reason - it's the auth query in your database server that is making things slow, accounting for more than half of the time it takes for your entire page load process to complete. Tracing can't tell you _why_ that's happening, but at least now you know where to look! + +`[kmclb: working w Sami on updating the illustrations to match this example. Keeping the link below just to have the example of how to format the markdown]` [{% asset performance/tracing-diagram.png alt="Diagram illustrating how a trace is composed of multiple transactions." %}]({% asset performance/tracing-diagram.png @path %}) ### Further Examples This section contains a few more examples of tracing, broken down into transactions and spans. -_Note:_ Starred spans represent spans that are the parent of a later transaction (and its root span). - **Measuring a Specific User Action** -If your application involves e-commerce, you likely want to measure the time between a user clicking "Submit Order" and the order confirmation appearing, including tracking the submitting of the charge to the payment processor and the sending of an order confirmation email. That entire process is one trace, and typically you'd have transactions for: - - - The browser's full process (example spans: XHR request to backend\*, rendering confirmation screen) - - Your backend's processing of that request (example spans: function call to compute total, DB call to store order*, API call to payment processor, queuing of email confirmation\*) - - Your database's work updating the customer's order history (example spans: individual SQL queries) - - The queued task of sending the email (example spans: function call to populate email template, API call to email-sending service) +If your application involves e-commerce, you likely want to measure the time between a user clicking "Submit Order" and the order confirmation appearing, including tracking the submitting of the charge to the payment processor and the sending of an order confirmation email. That entire process is one trace, and typically you'd have transactions (_T_) and spans (_S_) for: + +- The browser's full process (_T_) + - XHR request to backend\* (_S_) + - Rendering confirmation screen (_S_) +^ +- Your backend's processing of that request + - Function call to compute total (_S_) + - DB call to store order\* (_S_) + - API call to payment processor (_S_) + - Queuing of email confirmation\* (_S_) +^ +- Your database's work updating the customer's order history (_T_) + - Individual SQL queries (_S_) +^ +- The queued task of sending the email (_T_) + - Function call to populate email template (_S_) + - API call to email-sending service (_S_) -**Monitoring a Background Process** +_Note:_ Starred spans represent spans that are the parent of a later transaction (and its root span). -If your backend periodically polls for data from an external service, processes it, caches it, and then forwards it to an internal service, each instance of this happening is a trace, and you'd typically have transactions for: +**Monitoring a Background Process** -- The cron job that completes the entire process (example spans: API call to external service, processing function, call to caching service\*, API call to internal service\*) -- The work done in your caching service (example spans: checking cache for existing data, storing new data in cache) -- Your internal service's processing of the request (example spans: anything that service might do to handle the request) +If your backend periodically polls for data from an external service, processes it, caches it, and then forwards it to an internal service, each instance of this happening is a trace, and you'd typically have transactions (_T_) and spans (_S_) for: + +- The cron job that completes the entire process (_T_) + - API call to external service (_S_) + - Processing function (_S_) + - Call to caching service\* (_S_) + - API call to internal service\* (_S_) +^ +- The work done in your caching service (_T_) + - Checking cache for existing data (_S_) + - Storing new data in cache (_S_) +^ +- Your internal service's processing of the request (_T_) + - Anything that service might do to handle the request (_S_) + +_Note:_ Starred spans represent spans that are the parent of a later transaction (and its root span). ### Tracing Data Model -`[kmclb: do we want to format this in a quote-y kind of way? FPA: like a Note or Alert. I think it would be kind of cool.]` -"Show me your flowchart and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won't usually need your flowchart; it'll be obvious." -- [Fred Brooks](https://en.wikipedia.org/wiki/Fred_Brooks), The Mythical Man Month (1975) +> "Show me your flowchart and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won't usually need your flowchart; it'll be obvious." +> +> -- [Fred Brooks](https://en.wikipedia.org/wiki/Fred_Brooks), The Mythical Man Month (1975) While the theory is interesting, ultimately any data structure is defined by the kind of data it contains, and relationships between data structures are defined by how links between them are recorded. Traces, transactions, and spans are no different. `[FPA: this sentence is confusing to me. Do we mean. "Traces, transactions, and spans are defined by what's in them. The relationships among the three are defined by the manner in which the links between them are recorded." or something like that? kmclb: Does this read more easily to you?]` @@ -159,53 +177,58 @@ Traces are not an entity in and of themselves. Rather, a trace is defined as the **Transactions** -Transactions share most of their properties (start and end time, tags, and so forth) with their root spans, so the same options described below for spans are available in transactions, and setting them in either place is equivalent. `[FPA: suggest we say "so the same options described are available kmclb: A link felt like overkill for something which is 5 lines below, but hopefully the new wording makes it crystal clear which options we're talking about]` Transactions also have two additional properties not included in spans: - -`[kmclb: anything missing from this list?]` +Transactions share most of their properties (start and end time, tags, and so forth) with their root spans, so the same options described below for spans are available in transactions, and setting them in either place is equivalent. -- `transaction_name`: used in the UI to identify the transaction -- `root_span`: pointer to the root of the transactions span tree - -`[kmclb: how the heck do you get less-than or greater-than symbols to show up in code snippets? FPA: I usually just enclose in backticks, but it can be tricky - you can force also with html tags kmclb: how do you mean, enclose in backticks? How would you handle the user_id example below?]` -Common examples of `transaction_name` values include endpoint paths (like `/store/checkout/` or `api/v2/users/\/`) for backend request transactions, task names (like `data.cleanup.delete_inactive_users`) for cron job transactions, and URLs (like `https://docs.sentry.io/performance/distributed-tracing/`) for page-load transactions. +Transactions also have one additional property not included in spans, called `transaction_name`, which is used in the UI to identify the transaction. Common examples of `transaction_name` values include endpoint paths (like `/store/checkout/` or `api/v2/users/<user_id>/`) for backend request transactions, task names (like `data.cleanup.delete_inactive_users`) for cron job transactions, and URLs (like `https://docs.sentry.io/performance/distributed-tracing/`) for page-load transactions. **Spans** The majority of the data in a transaction resides in the individual spans the transaction contains. Span data includes: -`[kmclb: the list below isn't currently ground truth (quite), but the ground is potentially shifting. Will update once API is settled.]` - - - `parent_span_id`: ties the span to its parent span - - `op`: short code identifying the type of operation the span is measuring - - `start_timestamp`: when the span was opened - - `end_timestamp`: when the span was closed - - `description`: longer description of the span's operation, often specific to that instance (optional) - - `status`: short code indicating operation's status (optional) - - `tags`: key-value pairs holding additional data about the span (optional) - - `data`: arbitrarily-structured additional data about the span (optional) +- `parent_span_id`: ties the span to its parent span +- `op`: short string identifying the type of operation the span is measuring +- `start_timestamp`: when the span was opened +- `end_timestamp`: when the span was closed +- `description`: longer description of the span's operation, often specific to that instance (optional) +- `status`: short code indicating operation's status (optional) +- `tags`: key-value pairs holding additional data about the span (optional) +- `data`: arbitrarily-structured additional data about the span (optional) An example use of the `op` and `description` properties together is `op: sql.query` and `description: SELECT * FROM users WHERE last_active < DATE_SUB(CURRENT_DATE, INTERVAL 1 YEAR)`. The `status` property is often used to indicate the success or failure of the span's operation, or for a response code in the case of HTTP requests. Finally, `tags` and `data` allow you attach further contextual information to the span, such as `function: middleware.auth.is_authenticated` for a function call or `request: {url: ..., headers: ... , body: ...}` for an HTTP request. -### Good to Know (needs a better name) `[FPA: Tracing Relationships kmclb: but that's really what this entire (big) section is about. This smaller section is more "fun factoids which are helpful but didn't fit anywhere else" - some are about relationships between traces, transactions, and spans, but some are just about one of those]` +### Further Information A few more important points about traces, transactions, and spans, and the way they relate to one another: -**Trace Duration**: Because a trace just is a collection of transactions, traces don't have their own start and end times. Instead, a trace begins when its earliest transaction starts, and ends when its latest transaction ends. As a result, you can't explicitly start or end a trace directly. Instead, you create a trace by creating the first transaction in that trace, and you complete a trace by completing all of transactions it contains. +**Trace Duration** + +Because a trace just is a collection of transactions, traces don't have their own start and end times. Instead, a trace begins when its earliest transaction starts, and ends when its latest transaction ends. As a result, you can't explicitly start or end a trace directly. Instead, you create a trace by creating the first transaction in that trace, and you complete a trace by completing all of transactions it contains. + +**Async Transactions** + +Because of the possibility of asynchronous processes, child transactions may outlive the transactions containing their parent spans, sometimes by many orders of magnitude. For example, if a backend API call sets off a long-running processing task and then immediately returns a response, the backend transaction will finish (and its data will be sent to Sentry) long before the async task transaction does. Asynchronicity also means that the order in which transactions are sent to (and received by) Sentry does not in any way depend on the order in which they were created. (By contrast, order of receipt for transactions in the same trace _is_ correlated with order of completion, though because of factors like the variability of transmission times, the correlation is far from perfect.) + +**Orphan Transactions** -**Async Transactions**: Because of the possibility of asynchronous processes, child transactions may outlive the transactions containing their parent spans, sometimes by many orders of magnitude. For example, if a backend API call sets off a long-running processing task and then immediately returns a response, the backend transaction will finish (and its data will be sent to Sentry) long before the async task transaction does. Asynchronicity also means that the order in which transactions are sent to (and received by) Sentry does not in any way depend on the order in which they were created. (By contrast, order of receipt for transactions in the same trace _is_ correlated with order of completion, though because of factors like the variability of transmission times, the correlation is far from perfect.) +In theory, in a fully instrumented system, each trace should contain only one transaction and one span (the transaction's root) without a parent, namely the transaction in the originating service. However, in practice, you may not have tracing enabled in every one of your services, or an instrumented service may fail to report a transaction due to network disruption or other unforeseen circumstances. When this happens, you may see gaps in your trace hierarchy. Specifically, you may see transactions partway through the trace whose parent spans haven't been recorded as part of any known transactions. Such non-originating, parentless transactions are called **orphan transactions**. -**Orphan Transactions**: In theory, in a fully instrumented system, each trace should contain only one transaction and one span (the transaction's root) without a parent, namely the transaction in the originating service. However, in practice, you may not have tracing enabled in every one of your services, or an instrumented service may fail to report a transaction due to network disruption or other unforeseen circumstances. When this happens, you may see gaps in your trace hierarchy. Specifically, you may see transactions partway through the trace whose parent spans haven't been recorded as part of any known transactions. Such non-originating, parentless transactions are called **orphan transactions**. +**Nested Spans** -`[kmclb: depending on how we end up formatting this, may need to make this one paragraph]` -**Clock Skew**: If you are collecting transactions from multiple machines, you will likely encounter **clock skew**, where timestamps in one transaction don't align with timestamps in another. For example, if your backend makes a database call, the backend transaction logically should start before the database transaction does. But if the system time on each machine (those hosting your backend and database, respectively) isn't synced to common standard, it's possible that won't be the case. It's also possible for the ordering to be correct, but for the two recorded timeframes to not line up in a way that accurately reflects what actually happened. +Though our examples above had four levels in their hierarchy (trace, transaction, span, child span) there's no set limit to how deep the nesting of spans can go. There are, however, practical limits: transaction payloads sent to Sentry have a maximum allowed size, and as with any kind of logging, there's a balance to be struck between your data's granularity and its usability. -Because there is no way for Sentry to judge either the relative or absolute accuracy of your timestamps, it does not attempt to correct or modify them in any way. And while you can reduce clock skew by using Network Time Protocol (NTP) or your cloud provider's clock synchronization services, you may still notice small drifts in your data, as synchronizing clocks on small intervals is challenging. +**Zero-duration Spans** -**Nesting Spans**: Though our examples above had four levels in their hierarchy (trace, transaction, span, child span) there's no set limit to how deep the nesting of spans can go. There are, however, practical limits: transaction payloads sent to Sentry have a maximum allowed size (currently `[kmclb: find out what this is]`), and there's a balance to be struck between your data's granularity and its usability. +It's possible for a span to have equal start and end times, and therefore be recorded as taking no time. This can occur either because the span is being used as a marker (such as is done in [the browser's Performance API](https://developer.mozilla.org/en-US/docs/Web/API/Performance/mark)) or because the amount of time the operation took is less than the measurement resolution (which will vary by service). -**Zero-duration Spans**: It's possible for a span to have equal start and end times, and therefore be recorded as taking no time. This can occur either because the span is being used as a marker (such as is done in [the browser's Performance API](https://developer.mozilla.org/en-US/docs/Web/API/Performance/mark)) or because the amount of time the operation took is less than the measurement resolution (which will vary by service). +**Clock Skew** -**What We Send**: Individual spans aren't sent to Sentry. Rather, those spans are attached to their containing transaction, and the transaction is sent as one unit. This means that no span data will be recorded by Sentry's servers until the transaction to which they belong is closed and dispatched. +If you are collecting transactions from multiple machines, you will likely encounter **clock skew**, where timestamps in one transaction don't align with timestamps in another. For example, if your backend makes a database call, the backend transaction logically should start before the database transaction does. But if the system time on each machine (those hosting your backend and database, respectively) isn't synced to common standard, it's possible that won't be the case. It's also possible for the ordering to be correct, but for the two recorded timeframes to not line up in a way that accurately reflects what actually happened. + +Because there is no way for Sentry to judge either the relative or absolute accuracy of your timestamps, it does not attempt to correct for this. And while you can reduce clock skew by using Network Time Protocol (NTP) or your cloud provider's clock synchronization services, you may still notice small discrepancies in your data, as synchronizing clocks on small intervals is challenging. + +**What We Send** + +Individual spans aren't sent to Sentry. Rather, those spans are attached to their containing transaction, and the transaction is sent as one unit. This means that no span data will be recorded by Sentry's servers until the transaction to which they belong is closed and dispatched. ## Data Sampling @@ -215,7 +238,7 @@ When collecting traces, we **strongly recommend** sampling your data, for two re When choosing a sampling rate, the goal is to not collect _too_ much data (given the reasons above) but also to collect enough data that you are able to draw meaningful conclusions. If you're not sure what rate to choose, we recommend starting with a low value and gradually increasing it as you learn more about your traffic patterns and volume, until you've found a rate which lets you balance performance and cost concerns with data accuracy. -### Consistency Across Transactions Within a Trace +### Consistency Within a Trace For traces which involve multiple transactions, Sentry uses a "head-based" approach: a sampling decision is made in the originating service, and then that decision is passed to all subsequent services. To see how this works, let's return to our webapp example above. Consider two users, A and B, who are both loading your app in their respective browsers. When A loads the app, the SDK pseudorandomly "decides" to collect a trace, whereas when B loads the app, the SDK "decides" not to. When each browser makes requests to your backend, it includes in those requests the "yes, please collect transactions" or the "no, not collecting transactions this time" decision in the headers. @@ -225,9 +248,9 @@ On the other hand, when your backend processes the requests from B's browser, it Put simply: as a result of this head-based approach, where the decision is made once in the originating service and passed to all subsequent services, either all of the transactions for a given trace will be collected, or none will, so there shouldn't be any incomplete traces. -### Consistency Across Services +### Consistency Between Traces -If you enable tracing in services with multiple entry points, we recommend choosing similar sampling rates, to avoid biasing your data. For example, suppose the backend of our on-going webapp example also serves as a public API. In that case, some traces would start with a pageload transaction in the web app, and likely include multiple backend transactions, while other traces (those representing folks hitting the public API) would begin with a single backend request transaction, which would be the only backend transaction in the trace. Choosing a very different sampling rate for your webapp from that chosen for your backend would lead to one of those scenarios being oversampled compared to the other, compromising the accuracy of your overall data. +If you enable tracing in services with multiple entry points, we recommend choosing similar sampling rates, to avoid biasing your data. For example, suppose the backend of our on-going webapp example also serves as a public API. In that case, some traces would start with a pageload transaction in the web app, and likely include multiple backend transactions, while other traces (those representing folks hitting the public API) would begin with a single backend request transaction, which would be the only backend transaction in the trace. Choosing a very different sampling rate for your web app from that chosen for your backend would lead to one of those scenarios being oversampled compared to the other, compromising the accuracy of your overall data. ## Viewing Trace Data @@ -235,20 +258,18 @@ You can see a list of transaction events by clicking on the "Transactions" pre-b ### Transaction List View -`[kmclb: There's a LOT we don't document currently about the transaction list view (which is to say, almost anything). Some of it may be covered in the Discover docs, but even if so, I think we need to do a quick overview of all of the controls/what's displayed and then point to those docs. ]` +The results of either of the above queries will be presented in a list view, where each entry represents a group of one or more transactions. Data about each group is displayed in table form, and comes in two flavors: value-based (such as transaction name), and aggregate (such as average duration). The choice of which kinds of data to display is configurable, and can be changed by clicking 'Edit Columns' at the top right of the table. Bear in mind that adding or removing any value-based columns may affect the way the results are grouped. -`[kmclb: I think we should have a section - or at least a sentence or two - here on searching, which would mention that transactions (but not spans) are searchable, and what data one can use to search. Should also probably link to the main search docs.]` +From this view transactions can also be filtered, both by restricting the time window and by adding attributes to the query. It is important to note that **currently, only transaction data - the transaction name and any attributes the transaction inherits from its root span - is searchable. Data contained in spans other than the root span is not indexed and therefore is not searched.** -`[kmclb: We probably also want to explain how transactions are grouped in this view]` +Full documentation of the transaction list view (which is just a special case of the Discover query builder) can be found [here]({%- link _documentation/performance/discover/query-builder.md -%}). #### Performance Metrics `[kmclb: Help! This is a dangling modifier (there's a "you" implied by the first clause which never materializes in the second) but I can't think of a good way to fix it.]` When choosing which columns to display in your transaction list, there are a few metrics which lend themselves well to analyzing your application's performance. -`[kmclb: Do we want to mention the count() option? see note above above grouping]` - -**Aggregate Duration Metrics** +**Transaction Duration Metrics** You can aggregate transaction durations using the following functions: - average @@ -258,17 +279,17 @@ One use case for tracking these statistics is to help you identify transactions A word of caution when looking at averages and percentiles: In most cases, you'll want to set up tracing so that only [a fraction](#data-sampling) of possible traces are actually sent to Sentry, to avoid overwhelming your system. Further, you may want to filter your transaction data by date or other factors, or you may be tracing a relatively uncommon operation. For all of these reasons, you may end up with average and percentile data that is directionally correct, but not accurate. (To use the most extreme case as an example, if only a single transaction matches your filters, you can still compute an "average" duration, even though that's clearly not what is usually meant by "average.") -`[kmclb: Below: "We can calculate an average with less data than a 95th percentile" - is there some threshold below which we won't do the calculation?]` +The problem of small sample size (and the resulting inability to be usefully accurate) will happen more often for some metrics than others, and sample size will also vary by row. For example, it takes less data to calculate a meaningful average than it does to calculate an equally meaningful 95th percentile. Further, a row representing requests to `/settings/my-awesome-org/` will likely contain many times as many transactions as one representing requests to `/settings/my-awesome-org/projects/best-project-ever/`. -The problem of small sample size (and the resulting inability to be usefully accurate) will happen more often for some metrics than others. `[kmclb: which ones?]` We can calculate an average with less data than it takes to calculate the 95th percentile, but it’ll also vary by row. (For example, `/settings/` will always get more traffic than `/settings/country/belgium/tax`.) +**Transaction Frequency Metrics** -**Requests Per Minute (RPM)** +Transaction counts and the rate at which transactions are recorded can be calculated using the following functions: +- count +- count unique values (for a given field) +- average requests (transactions) per second +- average requests (transactions) per minute -`[kmclb: is there a reason we talk about RPM but not RPS?]` - -Requests Per Minute is a way to measure throughput. It is the average of all request durations, bucketed by the minute `[kmclb: start time or end time?]` for the current time window and query string. - -`[kmclb: now that I read this again, it doesn't make sense. Requests/min is a *number* of requests, whereas the description says it's the average *duration* of a group of requests. Which is it?]` +Each of these functions is calculated with respect to the collection of transactions within the given row, which means the numbers will change as you filter your data or change the time window. Also, you have set up your SDK to [sample your data](#data-sampling), remember that only the transactions which are sent to Sentry are counted. So if a row containing transactions representing requests to a given endpoint is calculated to be receiving 5 requests per second, and you've got a 25% sampling rate enabled, in reality you're getting approximately 20 (5 * 4) requests to that endpoint each second. ### Transaction Detail View @@ -300,16 +321,13 @@ Click on a row in the span view to expand the details of that span. From here, y You can search for all of the transactions in a given trace by expanding any of the span details and clicking on "Search by Trace". -_Note:_ Each transaction belongs to a specific project, and you will only be able to see transactions belonging to projects you have permission to view, which may or may not be all of the given trace's transactions. +_Note:_ On the Team plan, results will only be shown for one project at a time. Further, each transaction belongs to a specific project, and you will only be able to see transactions belonging to projects you have permission to view. Therefore, you may not see all transactions in a given trace in your results list. **Traversing to Child Transactions** Some spans within a transaction may be the parent of another transaction. When you expand the span details for such spans, you'll see the "View Child" button, which, when clicked, will lead to the child transaction's details view. -_Note:_ Each transaction belongs to a specific project, and you will only be able to see transactions belonging to projects you have permission to view, which may or may not include a given span's child transaction. `[kmclb: Does the button even show up if you don't have the right permissions?]` - - -`[kmclb: Haven't really touched much below this. It needs real work, but that's the next PR. :-) ]` +_Note:_ Traversing between transactions in this way is only available on the Business plan. Further, each transaction belongs to a specific project, and you will only be able to see the "View Child" button if the child transaction belongs to a project you have permission to view. ## Setting Up Tracing @@ -449,7 +467,6 @@ The default value of `tracingOrigins` is `['localhost', /^\//]`. The JavaScript - Therefore, the option needs to be configured like this: `new ApmIntegrations.Tracing({tracingOrigins: ['api.example.com']})` - Now outgoing XHR/fetch requests to `api.example.com` will get the `sentry-trace` header attached -`[kmclb: does this only apply to web servers, or all services which accept incoming requests?]` *NOTE:* You need to make sure your web server CORS is configured to allow the `sentry-trace` header. The configuration might look like `"Access-Control-Allow-Headers: sentry-trace"`, but this depends a lot on your set up. If you do not whitelist the `sentry-trace` header, the request might be blocked. **Using Tracing Integration for Manual Instrumentation** From d17a321668da54f831c716265c38bdc1851fdb9e Mon Sep 17 00:00:00 2001 From: Katie Byers Date: Thu, 7 May 2020 12:58:11 -0700 Subject: [PATCH 06/12] fiona edits --- .../performance/distributed-tracing.md | 35 ++++++++++--------- 1 file changed, 19 insertions(+), 16 deletions(-) diff --git a/src/collections/_documentation/performance/distributed-tracing.md b/src/collections/_documentation/performance/distributed-tracing.md index f9f0e42ff22c6..b6f59d0e954fa 100644 --- a/src/collections/_documentation/performance/distributed-tracing.md +++ b/src/collections/_documentation/performance/distributed-tracing.md @@ -169,7 +169,7 @@ _Note:_ Starred spans represent spans that are the parent of a later transaction > > -- [Fred Brooks](https://en.wikipedia.org/wiki/Fred_Brooks), The Mythical Man Month (1975) -While the theory is interesting, ultimately any data structure is defined by the kind of data it contains, and relationships between data structures are defined by how links between them are recorded. Traces, transactions, and spans are no different. `[FPA: this sentence is confusing to me. Do we mean. "Traces, transactions, and spans are defined by what's in them. The relationships among the three are defined by the manner in which the links between them are recorded." or something like that? kmclb: Does this read more easily to you?]` +While the theory is interesting, ultimately any data structure is defined by the kind of data it contains, and relationships between data structures are defined by how links between them are recorded. Traces, transactions, and spans are no different. **Traces** @@ -232,7 +232,7 @@ Individual spans aren't sent to Sentry. Rather, those spans are attached to thei ## Data Sampling -When you enable sampling in your tracing setup, you choose a percentage of collected transactions to send to Sentry. For example, if you had an endpoint that received 1000 requests per minute, a sampling rate of 0.25 would result in approximately 250 transactions (25%) being sent to Sentry. (The number is approximate because each request is either tracked or not, independently and pseudorandomly, with a 25% probability. So in the same way that 100 fair coins, when flipped, will result in approximately 50 heads, the SDK will decide to collect a trace in approximately 250 cases.) Because you know the sampling percentage, you can then extrapolate your total traffic volume. +When you enable sampling in your tracing setup, you choose a percentage of collected transactions to send to Sentry. For example, if you had an endpoint that received 1000 requests per minute, a sampling rate of 0.25 would result in approximately 250 transactions (25%) being sent to Sentry. (The number is approximate because each request is either tracked or not, independently and pseudorandomly, with a 25% probability. So in the same way that 100 fair coins, when flipped, result in approximately 50 heads, the SDK will "decide" to collect a trace in approximately 250 cases.) Because you know the sampling percentage, you can then extrapolate your total traffic volume. When collecting traces, we **strongly recommend** sampling your data, for two reasons. First, though capturing a single trace involves minimal overhead, capturing traces for every single pageload, or every single API request, has the potential to add an undesirable amount of load to your system. Second, by enabling sampling you'll more easily prevent yourself from exceeding your organization's [event quota]({%- link _documentation/accounts/quotas/index.md -%}), which will help you manage costs. @@ -240,13 +240,13 @@ When choosing a sampling rate, the goal is to not collect _too_ much data (given ### Consistency Within a Trace -For traces which involve multiple transactions, Sentry uses a "head-based" approach: a sampling decision is made in the originating service, and then that decision is passed to all subsequent services. To see how this works, let's return to our webapp example above. Consider two users, A and B, who are both loading your app in their respective browsers. When A loads the app, the SDK pseudorandomly "decides" to collect a trace, whereas when B loads the app, the SDK "decides" not to. When each browser makes requests to your backend, it includes in those requests the "yes, please collect transactions" or the "no, not collecting transactions this time" decision in the headers. +For traces that involve multiple transactions, Sentry uses a "head-based" approach: a sampling decision is made in the originating service, and then that decision is passed to all subsequent services. To see how this works, let's return to our webapp example above. Consider two users, A and B, who are both loading your app in their respective browsers. When A loads the app, the SDK pseudorandomly "decides" to collect a trace, whereas when B loads the app, the SDK "decides" not to. When each browser makes requests to your backend, it includes in those requests the "yes, please collect transactions" or the "no, not collecting transactions this time" decision in the headers. -When your backend processes the requests from A's browser, it will see the "yes" decision, collect transaction and span data, and send it to Sentry. Further, it will include the "yes" decision in any requests it makes to subsequent services (like your database server), which will similarly collect data, send it to Sentry, and pass the decision along to any services they call. Though this process, all of the relevant transactions in A's trace will be collected and sent to Sentry. +When your backend processes the requests from A's browser, it sees the "yes" decision, collects transaction and span data, and sends it to Sentry. Further, it includes the "yes" decision in any requests it makes to subsequent services (like your database server), which similarly collect data, send it to Sentry, and pass the decision along to any services they call. Through this process, all of the relevant transactions in A's trace are collected and sent to Sentry. -On the other hand, when your backend processes the requests from B's browser, it will see the "no" decision, and as a result it will _not_ collect and send transaction and span data to Sentry. It will, however, do the same thing it did in A's case in terms of propagating the decision to subsequent services, telling them not to collect or send data either. They will in turn tell any services they call not to send data, and in this way no transactions from B's trace will be collected. +On the other hand, when your backend processes the requests from B's browser, it sees the "no" decision, and as a result it does _not_ collect and send transaction and span data to Sentry. It does, however, do the same thing it does in A's case in terms of propagating the decision to subsequent services, telling them not to collect or send data either. They then in turn tell any services they call not to send data, and in this way no transactions from B's trace are collected. -Put simply: as a result of this head-based approach, where the decision is made once in the originating service and passed to all subsequent services, either all of the transactions for a given trace will be collected, or none will, so there shouldn't be any incomplete traces. +Put simply: as a result of this head-based approach, where the decision is made once in the originating service and passed to all subsequent services, either all of the transactions for a given trace are collected, or none are, so there shouldn't be any incomplete traces. ### Consistency Between Traces @@ -258,22 +258,24 @@ You can see a list of transaction events by clicking on the "Transactions" pre-b ### Transaction List View -The results of either of the above queries will be presented in a list view, where each entry represents a group of one or more transactions. Data about each group is displayed in table form, and comes in two flavors: value-based (such as transaction name), and aggregate (such as average duration). The choice of which kinds of data to display is configurable, and can be changed by clicking 'Edit Columns' at the top right of the table. Bear in mind that adding or removing any value-based columns may affect the way the results are grouped. +The results of either of the above queries are presented in a list view, where each entry represents a group of one or more transactions. Data about each group is displayed in table form, and comes in two flavors: value-based (such as transaction name), and aggregate (such as average duration). The choice of which kinds of data to display is configurable, and can be changed by clicking 'Edit Columns' at the top right of the table. Bear in mind that adding or removing any value-based columns may affect the way the results are grouped. -From this view transactions can also be filtered, both by restricting the time window and by adding attributes to the query. It is important to note that **currently, only transaction data - the transaction name and any attributes the transaction inherits from its root span - is searchable. Data contained in spans other than the root span is not indexed and therefore is not searched.** +From this view, you can also filter the transactions list, both by restricting the time window and by adding attributes to the query. + +_Note:_ Currently, only transaction data - the transaction name and any attributes the transaction inherits from its root span - is searchable. Data contained in spans other than the root span is not indexed and therefore is not searched. Full documentation of the transaction list view (which is just a special case of the Discover query builder) can be found [here]({%- link _documentation/performance/discover/query-builder.md -%}). #### Performance Metrics -`[kmclb: Help! This is a dangling modifier (there's a "you" implied by the first clause which never materializes in the second) but I can't think of a good way to fix it.]` -When choosing which columns to display in your transaction list, there are a few metrics which lend themselves well to analyzing your application's performance. +A number of the metrics available as column choices lend themselves well to analyzing your application's performance. **Transaction Duration Metrics** -You can aggregate transaction durations using the following functions: +The following functions aggregate transaction durations: + - average -- various percentiles (by default, the pre-built Transactions query shows the 75th and 95th percentiles, but there are many other options, including displaying a custom percentile) +- various percentiles (by default, the pre-built Transactions query shows the 75th and 95th percentiles, but there are many other options, including a custom percentile) One use case for tracking these statistics is to help you identify transactions that are slower than your organization's target SLAs. @@ -283,13 +285,14 @@ The problem of small sample size (and the resulting inability to be usefully acc **Transaction Frequency Metrics** -Transaction counts and the rate at which transactions are recorded can be calculated using the following functions: +The following functions aggregate transaction counts and the rate at which transactions are recorded: + - count - count unique values (for a given field) - average requests (transactions) per second - average requests (transactions) per minute -Each of these functions is calculated with respect to the collection of transactions within the given row, which means the numbers will change as you filter your data or change the time window. Also, you have set up your SDK to [sample your data](#data-sampling), remember that only the transactions which are sent to Sentry are counted. So if a row containing transactions representing requests to a given endpoint is calculated to be receiving 5 requests per second, and you've got a 25% sampling rate enabled, in reality you're getting approximately 20 (5 * 4) requests to that endpoint each second. +Each of these functions is calculated with respect to the collection of transactions within the given row, which means the numbers will change as you filter your data or change the time window. Also, if you have set up your SDK to [sample your data](#data-sampling), remember that only the transactions that are sent to Sentry are counted. So if a row containing transactions representing requests to a given endpoint is calculated to be receiving 5 requests per second, and you've got a 25% sampling rate enabled, in reality you're getting approximately 20 (5 * 4) requests to that endpoint each second. ### Transaction Detail View @@ -309,11 +312,11 @@ As shown in the Discover span screenshot above, you can click and drag your mous **Missing Instrumentation** -Sentry may indicate that gaps between spans are "Missing Instrumentation." This means that there is time in the transaction which isn't accounted for by any of the transaction's spans, and likely means that you need to manually instrument that part of your process. +Sentry may indicate that gaps between spans are "Missing Instrumentation." This means that there is time in the transaction that isn't accounted for by any of the transaction's spans, and likely means that you need to manually instrument that part of your process. **Viewing Span Details** -Click on a row in the span view to expand the details of that span. From here, you can see all attached properties, such as tags and data. +Clicking on a row in the span view to expands the details of that span. From here, you can see all attached properties, such as tags and data. [{% asset performance/span-detail-view.png alt="Span detail view shows the span id, trace id, parent span id, and other data such as tags." %}]({% asset performance/span-detail-view.png @path %}) From 79540f5d4d1e66fe2f44af48979c03c77fd72042 Mon Sep 17 00:00:00 2001 From: Katie Byers Date: Thu, 7 May 2020 14:21:14 -0700 Subject: [PATCH 07/12] update urls snapshot --- __tests__/__snapshots__/documentation.js.snap | 1 - 1 file changed, 1 deletion(-) diff --git a/__tests__/__snapshots__/documentation.js.snap b/__tests__/__snapshots__/documentation.js.snap index 027bdbd177066..187b7463c620c 100644 --- a/__tests__/__snapshots__/documentation.js.snap +++ b/__tests__/__snapshots__/documentation.js.snap @@ -299,7 +299,6 @@ Array [ "performance/discover/index.html", "performance/discover/query-builder/index.html", "performance/distributed-tracing/index.html", - "performance/performance-glossary/index.html", "platforms/android/index.html", "platforms/android/migrate/index.html", "platforms/cocoa/dsym/index.html", From e308ce391d81e30098505222c39cbb008dbf630d Mon Sep 17 00:00:00 2001 From: Katie Byers Date: Fri, 8 May 2020 16:55:13 -0700 Subject: [PATCH 08/12] incorporate daniel and rodolfo's edits --- .../performance/distributed-tracing.md | 52 +++++++++---------- 1 file changed, 26 insertions(+), 26 deletions(-) diff --git a/src/collections/_documentation/performance/distributed-tracing.md b/src/collections/_documentation/performance/distributed-tracing.md index b6f59d0e954fa..2b8d8fda4572f 100644 --- a/src/collections/_documentation/performance/distributed-tracing.md +++ b/src/collections/_documentation/performance/distributed-tracing.md @@ -24,7 +24,7 @@ To begin, a note about what tracing is not: Tracing is not profiling. Though the A [profiler](https://en.wikipedia.org/wiki/Profiling_(computer_programming)) may measure any number of aspects of an application's operation: the number of instructions executed, the amount of memory being used by various processes, the amount of time a given function call takes, and many more. The resulting profile is a statistical summary of these measurements. -A [tracing tool](https://en.wikipedia.org/wiki/Tracing_(software)), on the other hand, focuses on _what_ happened (and when), rather than how many times it happened or how long it took. The resulting trace is log of events which occurred during a program's execution, often across multiple systems. Though traces most often - or, in the case of Sentry's traces, always - include timestamps, allowing durations to be calculated, measuring performance is not their only purpose. They can also show the ways in which interconnected systems interact, and the ways in which problems in one can cause problems in another. +A [tracing tool](https://en.wikipedia.org/wiki/Tracing_(software)), on the other hand, focuses on _what_ happened (and when), rather than how many times it happened or how long it took. The resulting trace is a log of events which occurred during a program's execution, often across multiple systems. Though traces most often - or, in the case of Sentry's traces, always - include timestamps, allowing durations to be calculated, measuring performance is not their only purpose. They can also show the ways in which interconnected systems interact, and the ways in which problems in one can cause problems in another. ### Why Tracing? @@ -54,7 +54,7 @@ To make all of this more concrete, let's consider our example web app again. ### Example: Investigating Slow Page Load -Suppose your web application is slow to load, and you'd like to know why. A lot has to happen for your app to first get to a usable state: multiple requests to your backend, likely some work - including calls to your database or to outside APis - completed before responses are returned, and processing by the browser to render all of the returned data into something meaningful to the user. So which part of that process is slowing things down? +Suppose your web application is slow to load, and you'd like to know why. A lot has to happen for your app to first get to a usable state: multiple requests to your backend, likely some work - including calls to your database or to outside APIs - completed before responses are returned, and processing by the browser to render all of the returned data into something meaningful to the user. So which part of that process is slowing things down? Let's say, in this simplified example, that when a user loads the app in their browser, the following happens in each service: @@ -91,7 +91,7 @@ Each transaction would be broken down into **spans** as follows: Let's pause here to make an important point: Some, though not all, of the browser transaction spans listed have a direct correspondence to backend transactions listed earlier. Specifically, each request _span_ in the browser transaction corresponds to a separate request _transaction_ in the backend. In this situation, when a span in one service gives rise to a transaction in a subsequent service, we call the original span a parent span to _both_ the transaction and its root span. -In our example, every transaction other than the initial browser page-load transaction is the child of a span in another service, which means that every root span other than the browser transaction root has a parent span (albeit in a different service). In a fully-instrumented system (one in which every service has tracing enabled) this pattern will always hold true: _the only parentless span will be the root of the initial transaction, and of the remaining parented spans, every one will live in the same service as its parent, except for the root spans, whose parents will live in a previous service_. This is worth noting because it is the one way in which transactions are not perfect trees - their roots can (and mostly do) have parents. +In our example, every transaction other than the initial browser page-load transaction is the child of a span in another service, which means that every root span other than the browser transaction root has a parent span (albeit in a different service). In a fully-instrumented system (one in which every service has tracing enabled) this pattern will always hold true. The only parentless span will be the root of the initial transaction; every other span will have a parent. Further, parents and children will always live in the same service, except in the case where the child span is the root of a child transaction, in which case the parent span will live in the calling service and the child transaction/child root span will live in the called service. (This is worth noting because it is the one way in which transactions are not perfect trees - their roots can (and mostly do) have parents.) Now, for the sake of completeness, back to our spans: @@ -114,8 +114,8 @@ Now, for the sake of completeness, back to our spans: To wrap up the example: after instrumenting all of your services, you might discover that - for some reason - it's the auth query in your database server that is making things slow, accounting for more than half of the time it takes for your entire page load process to complete. Tracing can't tell you _why_ that's happening, but at least now you know where to look! -`[kmclb: working w Sami on updating the illustrations to match this example. Keeping the link below just to have the example of how to format the markdown]` -[{% asset performance/tracing-diagram.png alt="Diagram illustrating how a trace is composed of multiple transactions." %}]({% asset performance/tracing-diagram.png @path %}) + ### Further Examples @@ -125,20 +125,20 @@ This section contains a few more examples of tracing, broken down into transacti If your application involves e-commerce, you likely want to measure the time between a user clicking "Submit Order" and the order confirmation appearing, including tracking the submitting of the charge to the payment processor and the sending of an order confirmation email. That entire process is one trace, and typically you'd have transactions (_T_) and spans (_S_) for: -- The browser's full process (_T_) +- The browser's full process (_T_ and root span _S_) - XHR request to backend\* (_S_) - Rendering confirmation screen (_S_) ^ -- Your backend's processing of that request +- Your backend's processing of that request (_T_ and root span _S_) - Function call to compute total (_S_) - DB call to store order\* (_S_) - API call to payment processor (_S_) - Queuing of email confirmation\* (_S_) ^ -- Your database's work updating the customer's order history (_T_) +- Your database's work updating the customer's order history (_T_ and root span _S_) - Individual SQL queries (_S_) ^ -- The queued task of sending the email (_T_) +- The queued task of sending the email (_T_ and root span _S_) - Function call to populate email template (_S_) - API call to email-sending service (_S_) @@ -148,17 +148,17 @@ _Note:_ Starred spans represent spans that are the parent of a later transaction If your backend periodically polls for data from an external service, processes it, caches it, and then forwards it to an internal service, each instance of this happening is a trace, and you'd typically have transactions (_T_) and spans (_S_) for: -- The cron job that completes the entire process (_T_) +- The cron job that completes the entire process (_T_ and root span _S_) - API call to external service (_S_) - Processing function (_S_) - Call to caching service\* (_S_) - API call to internal service\* (_S_) ^ -- The work done in your caching service (_T_) +- The work done in your caching service (_T_ and root span _S_) - Checking cache for existing data (_S_) - Storing new data in cache (_S_) ^ -- Your internal service's processing of the request (_T_) +- Your internal service's processing of the request (_T_ and root span _S_) - Anything that service might do to handle the request (_S_) _Note:_ Starred spans represent spans that are the parent of a later transaction (and its root span). @@ -194,7 +194,7 @@ The majority of the data in a transaction resides in the individual spans the tr - `tags`: key-value pairs holding additional data about the span (optional) - `data`: arbitrarily-structured additional data about the span (optional) -An example use of the `op` and `description` properties together is `op: sql.query` and `description: SELECT * FROM users WHERE last_active < DATE_SUB(CURRENT_DATE, INTERVAL 1 YEAR)`. The `status` property is often used to indicate the success or failure of the span's operation, or for a response code in the case of HTTP requests. Finally, `tags` and `data` allow you attach further contextual information to the span, such as `function: middleware.auth.is_authenticated` for a function call or `request: {url: ..., headers: ... , body: ...}` for an HTTP request. +An example use of the `op` and `description` properties together is `op: sql.query` and `description: SELECT * FROM users WHERE last_active < DATE_SUB(CURRENT_DATE, INTERVAL 1 YEAR)`. The `status` property is often used to indicate the success or failure of the span's operation, or for a response code in the case of HTTP requests. Finally, `tags` and `data` allow you to attach further contextual information to the span, such as `function: middleware.auth.is_authenticated` for a function call or `request: {url: ..., headers: ... , body: ...}` for an HTTP request. ### Further Information @@ -226,21 +226,21 @@ If you are collecting transactions from multiple machines, you will likely encou Because there is no way for Sentry to judge either the relative or absolute accuracy of your timestamps, it does not attempt to correct for this. And while you can reduce clock skew by using Network Time Protocol (NTP) or your cloud provider's clock synchronization services, you may still notice small discrepancies in your data, as synchronizing clocks on small intervals is challenging. -**What We Send** +**How Data is Sent** -Individual spans aren't sent to Sentry. Rather, those spans are attached to their containing transaction, and the transaction is sent as one unit. This means that no span data will be recorded by Sentry's servers until the transaction to which they belong is closed and dispatched. +Individual spans aren't sent to Sentry. Rather, those spans are attached to their containing transaction, and the transaction is sent as one unit. This means that no span data will be recorded by Sentry's servers until the transaction to which they belong is closed and dispatched. The converse is not true, however - though spans can't be sent without a transaction, transaction _are_ still valid, and will be sent, even if they only span they contain is their root span. ## Data Sampling -When you enable sampling in your tracing setup, you choose a percentage of collected transactions to send to Sentry. For example, if you had an endpoint that received 1000 requests per minute, a sampling rate of 0.25 would result in approximately 250 transactions (25%) being sent to Sentry. (The number is approximate because each request is either tracked or not, independently and pseudorandomly, with a 25% probability. So in the same way that 100 fair coins, when flipped, result in approximately 50 heads, the SDK will "decide" to collect a trace in approximately 250 cases.) Because you know the sampling percentage, you can then extrapolate your total traffic volume. +When you enable sampling in your tracing setup, you choose a percentage of collected transactions to send to Sentry. For example, if you had an endpoint that received 1000 requests per minute, a sampling rate of `0.25` would result in approximately 250 transactions (25%) being sent to Sentry each minute. (The number is approximate because each request is either tracked or not, independently and pseudorandomly, with a 25% probability. So in the same way that 100 fair coins, when flipped, result in approximately 50 heads, the SDK will "decide" to collect a trace in approximately 250 cases.) Because you know the sampling percentage, you can then extrapolate your total traffic volume. -When collecting traces, we **strongly recommend** sampling your data, for two reasons. First, though capturing a single trace involves minimal overhead, capturing traces for every single pageload, or every single API request, has the potential to add an undesirable amount of load to your system. Second, by enabling sampling you'll more easily prevent yourself from exceeding your organization's [event quota]({%- link _documentation/accounts/quotas/index.md -%}), which will help you manage costs. +When collecting traces, we **strongly recommend** sampling your data, for two reasons. First, though capturing a single trace involves minimal overhead, capturing traces for every single page load, or every single API request, has the potential to add an undesirable amount of load to your system. Second, by enabling sampling you'll more easily prevent yourself from exceeding your organization's [event quota]({%- link _documentation/accounts/quotas/index.md -%}), which will help you manage costs. When choosing a sampling rate, the goal is to not collect _too_ much data (given the reasons above) but also to collect enough data that you are able to draw meaningful conclusions. If you're not sure what rate to choose, we recommend starting with a low value and gradually increasing it as you learn more about your traffic patterns and volume, until you've found a rate which lets you balance performance and cost concerns with data accuracy. ### Consistency Within a Trace -For traces that involve multiple transactions, Sentry uses a "head-based" approach: a sampling decision is made in the originating service, and then that decision is passed to all subsequent services. To see how this works, let's return to our webapp example above. Consider two users, A and B, who are both loading your app in their respective browsers. When A loads the app, the SDK pseudorandomly "decides" to collect a trace, whereas when B loads the app, the SDK "decides" not to. When each browser makes requests to your backend, it includes in those requests the "yes, please collect transactions" or the "no, not collecting transactions this time" decision in the headers. +For traces that involve multiple transactions, Sentry uses a "head-based" approach: a sampling decision is made in the originating service, and then that decision is passed to all subsequent services. To see how this works, let's return to our web app example above. Consider two users, A and B, who are both loading your app in their respective browsers. When A loads the app, the SDK pseudorandomly "decides" to collect a trace, whereas when B loads the app, the SDK "decides" not to. When each browser makes requests to your backend, it includes in those requests the "yes, please collect transactions" or the "no, not collecting transactions this time" decision in the headers. When your backend processes the requests from A's browser, it sees the "yes" decision, collects transaction and span data, and sends it to Sentry. Further, it includes the "yes" decision in any requests it makes to subsequent services (like your database server), which similarly collect data, send it to Sentry, and pass the decision along to any services they call. Through this process, all of the relevant transactions in A's trace are collected and sent to Sentry. @@ -250,7 +250,7 @@ Put simply: as a result of this head-based approach, where the decision is made ### Consistency Between Traces -If you enable tracing in services with multiple entry points, we recommend choosing similar sampling rates, to avoid biasing your data. For example, suppose the backend of our on-going webapp example also serves as a public API. In that case, some traces would start with a pageload transaction in the web app, and likely include multiple backend transactions, while other traces (those representing folks hitting the public API) would begin with a single backend request transaction, which would be the only backend transaction in the trace. Choosing a very different sampling rate for your web app from that chosen for your backend would lead to one of those scenarios being oversampled compared to the other, compromising the accuracy of your overall data. +If you enable tracing in services with multiple entry points, we recommend choosing similar sampling rates, to avoid biasing your data. For example, suppose the backend of our on-going web app example also serves as a public API. In that case, some traces would start with a page-load transaction in the web app, and likely include multiple backend transactions, while other traces (those representing folks hitting the public API) would begin with a single backend request transaction, which would be the only backend transaction in the trace. Choosing a very different sampling rate for your web app from that chosen for your backend would lead to one of those scenarios being oversampled compared to the other, compromising the accuracy of your overall data. ## Viewing Trace Data @@ -262,9 +262,9 @@ The results of either of the above queries are presented in a list view, where e From this view, you can also filter the transactions list, both by restricting the time window and by adding attributes to the query. -_Note:_ Currently, only transaction data - the transaction name and any attributes the transaction inherits from its root span - is searchable. Data contained in spans other than the root span is not indexed and therefore is not searched. +_Note:_ Currently, only transaction data - the transaction name and any attributes the transaction inherits from its root span - is searchable. Data contained in spans other than the root span is not indexed and therefore cannot be searched. -Full documentation of the transaction list view (which is just a special case of the Discover query builder) can be found [here]({%- link _documentation/performance/discover/query-builder.md -%}). +Full documentation of the transaction list view (which is just a special case of the Discover Query Builder) can be found [here]({%- link _documentation/performance/discover/query-builder.md -%}). #### Performance Metrics @@ -292,7 +292,7 @@ The following functions aggregate transaction counts and the rate at which trans - average requests (transactions) per second - average requests (transactions) per minute -Each of these functions is calculated with respect to the collection of transactions within the given row, which means the numbers will change as you filter your data or change the time window. Also, if you have set up your SDK to [sample your data](#data-sampling), remember that only the transactions that are sent to Sentry are counted. So if a row containing transactions representing requests to a given endpoint is calculated to be receiving 5 requests per second, and you've got a 25% sampling rate enabled, in reality you're getting approximately 20 (5 * 4) requests to that endpoint each second. +Each of these functions is calculated with respect to the collection of transactions within the given row, which means the numbers will change as you filter your data or change the time window. Also, if you have set up your SDK to [sample your data](#data-sampling), remember that only the transactions that are sent to Sentry are counted. So if a row containing transactions representing requests to a given endpoint is calculated to be receiving 5 requests per second, and you've got a 25% sampling rate enabled, in reality you're getting approximately 20 requests to that endpoint each second. (20 because you're sampling 25% - or 1/4 - of your data, so your real volume is 4 times what you're seeing in Sentry.) ### Transaction Detail View @@ -316,7 +316,7 @@ Sentry may indicate that gaps between spans are "Missing Instrumentation." This **Viewing Span Details** -Clicking on a row in the span view to expands the details of that span. From here, you can see all attached properties, such as tags and data. +Clicking on a row in the span view expands the details of that span. From here, you can see all attached properties, such as tags and data. [{% asset performance/span-detail-view.png alt="Span detail view shows the span id, trace id, parent span id, and other data such as tags." %}]({% asset performance/span-detail-view.png @path %}) @@ -364,7 +364,7 @@ sentry_sdk.init( **Automatic Instrumentation** -Many integrations for popular frameworks automatically capture traces. If you already have any of the following frameworks set up for Sentry error reporting, you will start to see traces immediately: +Many integrations for popular frameworks automatically capture transactions. If you already have any of the following frameworks set up for Sentry error reporting, you will start to see traces immediately: - All WSGI-based web frameworks (Django, Flask, Pyramid, Falcon, Bottle) - Celery @@ -452,7 +452,7 @@ Sentry.init({ To send traces, you will need to set the `tracesSampleRate` to a nonzero value. The configuration above will capture 25% of your transactions. -You can pass many different options to the Tracing integration (as an object of the form `{optionName: value}`), but it comes with reasonable defaults out of the box. It will automatically capture a trace for every page load. Within that transaction, spans are instrumented for the following operations: +You can pass many different options to the Tracing integration (as an object of the form `{optionName: value}`), but it comes with reasonable defaults out of the box. It will automatically capture a transaction for every page load. Within that transaction, spans are instrumented for the following operations: - XHR/fetch requests - If available: Browser Resources (Scripts, CSS, Images ...) @@ -474,7 +474,7 @@ The default value of `tracingOrigins` is `['localhost', /^\//]`. The JavaScript **Using Tracing Integration for Manual Instrumentation** -The tracing integration will create a transaction on page load by default; all spans that are created will be attached to it. Also, the integration will finish the transaction after the default timeout of 500ms of inactivity. The page is considered inactive if there are no pending XHR/fetch requests. If you want to extend the transaction's lifetime, you can do so by adding more spans to it. The following is an example of how you could profile a React component: +The tracing integration will create a transaction on page load by default; all spans that are created will be attached to it. Also, the integration will finish the transaction after the default timeout of 500ms of inactivity. The page is considered inactive if there are no pending XHR/fetch requests. If you want to extend the transaction's lifetime, you can do so by adding more spans to it. The following is an example of how you could manually instrument a React component: ```javascript // This line starts an activity (and creates a span). From d51d8a58a5c63fd3a053f456395dcecf9cd03b2f Mon Sep 17 00:00:00 2001 From: Katie Byers Date: Fri, 8 May 2020 17:36:15 -0700 Subject: [PATCH 09/12] switch to using real headings --- .../performance/distributed-tracing.md | 66 +++++++++---------- 1 file changed, 32 insertions(+), 34 deletions(-) diff --git a/src/collections/_documentation/performance/distributed-tracing.md b/src/collections/_documentation/performance/distributed-tracing.md index 2b8d8fda4572f..677461e282895 100644 --- a/src/collections/_documentation/performance/distributed-tracing.md +++ b/src/collections/_documentation/performance/distributed-tracing.md @@ -121,7 +121,7 @@ To wrap up the example: after instrumenting all of your services, you might disc This section contains a few more examples of tracing, broken down into transactions and spans. -**Measuring a Specific User Action** +#### Measuring a Specific User Action If your application involves e-commerce, you likely want to measure the time between a user clicking "Submit Order" and the order confirmation appearing, including tracking the submitting of the charge to the payment processor and the sending of an order confirmation email. That entire process is one trace, and typically you'd have transactions (_T_) and spans (_S_) for: @@ -144,7 +144,7 @@ If your application involves e-commerce, you likely want to measure the time bet _Note:_ Starred spans represent spans that are the parent of a later transaction (and its root span). -**Monitoring a Background Process** +#### Monitoring a Background Process If your backend periodically polls for data from an external service, processes it, caches it, and then forwards it to an internal service, each instance of this happening is a trace, and you'd typically have transactions (_T_) and spans (_S_) for: @@ -171,17 +171,17 @@ _Note:_ Starred spans represent spans that are the parent of a later transaction While the theory is interesting, ultimately any data structure is defined by the kind of data it contains, and relationships between data structures are defined by how links between them are recorded. Traces, transactions, and spans are no different. -**Traces** +#### Traces Traces are not an entity in and of themselves. Rather, a trace is defined as the collection of all transactions that share a `trace_id` value. -**Transactions** +#### Transactions Transactions share most of their properties (start and end time, tags, and so forth) with their root spans, so the same options described below for spans are available in transactions, and setting them in either place is equivalent. Transactions also have one additional property not included in spans, called `transaction_name`, which is used in the UI to identify the transaction. Common examples of `transaction_name` values include endpoint paths (like `/store/checkout/` or `api/v2/users/<user_id>/`) for backend request transactions, task names (like `data.cleanup.delete_inactive_users`) for cron job transactions, and URLs (like `https://docs.sentry.io/performance/distributed-tracing/`) for page-load transactions. -**Spans** +#### Spans The majority of the data in a transaction resides in the individual spans the transaction contains. Span data includes: @@ -200,33 +200,31 @@ An example use of the `op` and `description` properties together is `op: sql.que A few more important points about traces, transactions, and spans, and the way they relate to one another: -**Trace Duration** +#### Trace Duration Because a trace just is a collection of transactions, traces don't have their own start and end times. Instead, a trace begins when its earliest transaction starts, and ends when its latest transaction ends. As a result, you can't explicitly start or end a trace directly. Instead, you create a trace by creating the first transaction in that trace, and you complete a trace by completing all of transactions it contains. -**Async Transactions** +#### Async Transactions Because of the possibility of asynchronous processes, child transactions may outlive the transactions containing their parent spans, sometimes by many orders of magnitude. For example, if a backend API call sets off a long-running processing task and then immediately returns a response, the backend transaction will finish (and its data will be sent to Sentry) long before the async task transaction does. Asynchronicity also means that the order in which transactions are sent to (and received by) Sentry does not in any way depend on the order in which they were created. (By contrast, order of receipt for transactions in the same trace _is_ correlated with order of completion, though because of factors like the variability of transmission times, the correlation is far from perfect.) -**Orphan Transactions** +#### Orphan Transactions In theory, in a fully instrumented system, each trace should contain only one transaction and one span (the transaction's root) without a parent, namely the transaction in the originating service. However, in practice, you may not have tracing enabled in every one of your services, or an instrumented service may fail to report a transaction due to network disruption or other unforeseen circumstances. When this happens, you may see gaps in your trace hierarchy. Specifically, you may see transactions partway through the trace whose parent spans haven't been recorded as part of any known transactions. Such non-originating, parentless transactions are called **orphan transactions**. -**Nested Spans** +#### Nested Spans Though our examples above had four levels in their hierarchy (trace, transaction, span, child span) there's no set limit to how deep the nesting of spans can go. There are, however, practical limits: transaction payloads sent to Sentry have a maximum allowed size, and as with any kind of logging, there's a balance to be struck between your data's granularity and its usability. -**Zero-duration Spans** +#### Zero-duration Spans It's possible for a span to have equal start and end times, and therefore be recorded as taking no time. This can occur either because the span is being used as a marker (such as is done in [the browser's Performance API](https://developer.mozilla.org/en-US/docs/Web/API/Performance/mark)) or because the amount of time the operation took is less than the measurement resolution (which will vary by service). -**Clock Skew** +#### Clock Skew -If you are collecting transactions from multiple machines, you will likely encounter **clock skew**, where timestamps in one transaction don't align with timestamps in another. For example, if your backend makes a database call, the backend transaction logically should start before the database transaction does. But if the system time on each machine (those hosting your backend and database, respectively) isn't synced to common standard, it's possible that won't be the case. It's also possible for the ordering to be correct, but for the two recorded timeframes to not line up in a way that accurately reflects what actually happened. +If you are collecting transactions from multiple machines, you will likely encounter **clock skew**, where timestamps in one transaction don't align with timestamps in another. For example, if your backend makes a database call, the backend transaction logically should start before the database transaction does. But if the system time on each machine (those hosting your backend and database, respectively) isn't synced to common standard, it's possible that won't be the case. It's also possible for the ordering to be correct, but for the two recorded timeframes to not line up in a way that accurately reflects what actually happened. To reduce this possibility, we recommend using Network Time Protocol (NTP) or your cloud provider's clock synchronization services. -Because there is no way for Sentry to judge either the relative or absolute accuracy of your timestamps, it does not attempt to correct for this. And while you can reduce clock skew by using Network Time Protocol (NTP) or your cloud provider's clock synchronization services, you may still notice small discrepancies in your data, as synchronizing clocks on small intervals is challenging. - -**How Data is Sent** +#### How Data is Sent Individual spans aren't sent to Sentry. Rather, those spans are attached to their containing transaction, and the transaction is sent as one unit. This means that no span data will be recorded by Sentry's servers until the transaction to which they belong is closed and dispatched. The converse is not true, however - though spans can't be sent without a transaction, transaction _are_ still valid, and will be sent, even if they only span they contain is their root span. @@ -270,7 +268,7 @@ Full documentation of the transaction list view (which is just a special case of A number of the metrics available as column choices lend themselves well to analyzing your application's performance. -**Transaction Duration Metrics** +_Transaction Duration Metrics_ The following functions aggregate transaction durations: @@ -283,7 +281,7 @@ A word of caution when looking at averages and percentiles: In most cases, you'l The problem of small sample size (and the resulting inability to be usefully accurate) will happen more often for some metrics than others, and sample size will also vary by row. For example, it takes less data to calculate a meaningful average than it does to calculate an equally meaningful 95th percentile. Further, a row representing requests to `/settings/my-awesome-org/` will likely contain many times as many transactions as one representing requests to `/settings/my-awesome-org/projects/best-project-ever/`. -**Transaction Frequency Metrics** +_Transaction Frequency Metrics_ The following functions aggregate transaction counts and the rate at which transactions are recorded: @@ -306,27 +304,27 @@ The span view is a split view where the left-hand side shows the transaction's s At the top of the span view is a minimap, which shows which specific portion of the transaction you're viewing. -**Zooming In on a Transaction** +_Zooming In on a Transaction_ As shown in the Discover span screenshot above, you can click and drag your mouse cursor across the minimap (top of the span view). You can also adjust the window selection by dragging the handlebars (black dashed lines). -**Missing Instrumentation** +_Missing Instrumentation_ Sentry may indicate that gaps between spans are "Missing Instrumentation." This means that there is time in the transaction that isn't accounted for by any of the transaction's spans, and likely means that you need to manually instrument that part of your process. -**Viewing Span Details** +_Viewing Span Details_ Clicking on a row in the span view expands the details of that span. From here, you can see all attached properties, such as tags and data. [{% asset performance/span-detail-view.png alt="Span detail view shows the span id, trace id, parent span id, and other data such as tags." %}]({% asset performance/span-detail-view.png @path %}) -**Search by Trace ID** +_Search by Trace ID_ You can search for all of the transactions in a given trace by expanding any of the span details and clicking on "Search by Trace". _Note:_ On the Team plan, results will only be shown for one project at a time. Further, each transaction belongs to a specific project, and you will only be able to see transactions belonging to projects you have permission to view. Therefore, you may not see all transactions in a given trace in your results list. -**Traversing to Child Transactions** +_Traversing to Child Transactions_ Some spans within a transaction may be the parent of another transaction. When you expand the span details for such spans, you'll see the "View Child" button, which, when clicked, will lead to the child transaction's details view. @@ -362,7 +360,7 @@ sentry_sdk.init( ) ``` -**Automatic Instrumentation** +#### Automatic Instrumentation Many integrations for popular frameworks automatically capture transactions. If you already have any of the following frameworks set up for Sentry error reporting, you will start to see traces immediately: @@ -389,7 +387,7 @@ sentry_sdk.init( ) ``` -**Manual Instrumentation** +#### Manual Instrumentation To manually instrument certain regions of your code, you can create a transaction to capture them. @@ -406,7 +404,7 @@ while True: process_item(item) ``` -**Adding More Spans to the Transaction** +#### Adding More Spans to the Transaction The next example contains the implementation of the hypothetical `process_item` function called from the code snippet in the previous section. Our SDK can determine if there is currently an open transaction and add all newly created spans as child operations to that transaction. Keep in mind that each individual span also needs to be manually finished; otherwise, spans will not show up in the transaction. @@ -433,7 +431,7 @@ $ npm install @sentry/browser $ npm install @sentry/apm ``` -**Sending Traces/Transactions/Spans** +#### Sending Traces/Transactions/Spans The `Tracing` integration resides in the `@sentry/apm` package. You can add it to your `Sentry.init` call: @@ -472,7 +470,7 @@ The default value of `tracingOrigins` is `['localhost', /^\//]`. The JavaScript *NOTE:* You need to make sure your web server CORS is configured to allow the `sentry-trace` header. The configuration might look like `"Access-Control-Allow-Headers: sentry-trace"`, but this depends a lot on your set up. If you do not whitelist the `sentry-trace` header, the request might be blocked. -**Using Tracing Integration for Manual Instrumentation** +#### Using Tracing Integration for Manual Instrumentation The tracing integration will create a transaction on page load by default; all spans that are created will be attached to it. Also, the integration will finish the transaction after the default timeout of 500ms of inactivity. The page is considered inactive if there are no pending XHR/fetch requests. If you want to extend the transaction's lifetime, you can do so by adding more spans to it. The following is an example of how you could manually instrument a React component: @@ -527,11 +525,11 @@ shopCheckout() { This example will send a transaction `shopCheckout` to Sentry, containing all outgoing requests that happen in `validateShoppingCartOnServer` as spans. The transaction will also contain a `task` span that measures how long `processAndValidateShoppingCart` took. Finally, the call to `ApmIntegrations.Tracing.finishIdleTransaction()` will finish the transaction and send it to Sentry. Calling this is optional; if it is not called, the integration will automatically send the transaction itself after the defined `idleTimeout` (default 500ms). -**What is an activity?** +#### What is an activity? The concept of an activity only exists in the `Tracing` integration in JavaScript Browser. It's a helper that tells the integration how long it should keep the `IdleTransaction` alive. An activity can be pushed and popped. Once all activities of an `IdleTransaction` have been popped, the `IdleTransaction` will be sent to Sentry. -**Manual Transactions** +#### Manual Transactions To manually instrument a certain region of your code, you can create a transaction to capture it. This is valid for both JavaScript Browser and Node and works independent of the `Tracing` integration. @@ -551,7 +549,7 @@ processItem(item).then(() => { }) ``` -**Adding Additional Spans to the Transaction** +#### Adding Additional Spans to the Transaction The next example contains the implementation of the hypothetical `processItem ` function called from the code snippet in the previous section. Our SDK can determine if there is currently an open transaction and add to it all newly created spans as child operations. Keep in mind that each individual span needs to be manually finished; otherwise, that span will not show up in the transaction. @@ -585,7 +583,7 @@ $ npm install @sentry/node $ npm install @sentry/apm ``` -**Sending Traces** +#### Sending Traces To send traces, set the `tracesSampleRate` to a nonzero value. The following configuration will capture 25% of all your transactions: @@ -601,7 +599,7 @@ Sentry.init({ }); ``` -**Automatic Instrumentation** +#### Automatic Instrumentation It’s possible to add tracing to all popular frameworks; however, we provide pre-written handlers only for Express.js. @@ -654,7 +652,7 @@ Sentry.init({ }); ``` -**Manual Transactions** +#### Manual Transactions To manually instrument a certain region of your code, you can create a transaction to capture it. @@ -676,7 +674,7 @@ app.use(function processItems(req, res, next) { }); ``` -**Adding Additional Spans to the Transaction** +#### Adding Additional Spans to the Transaction The next example contains the implementation of the hypothetical `processItem ` function called from the code snippet in the previous section. Our SDK can determine if there is currently an open transaction and add to it all newly created spans as child operations. Keep in mind that each individual span needs to be manually finished; otherwise, that span will not show up in the transaction. From d73aa5d8a9ba730396a30a3f74aeb104c6948ca0 Mon Sep 17 00:00:00 2001 From: Katie Byers Date: Mon, 11 May 2020 12:56:47 -0700 Subject: [PATCH 10/12] more friday edits --- .../performance/distributed-tracing.md | 27 ++++++++++--------- 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/src/collections/_documentation/performance/distributed-tracing.md b/src/collections/_documentation/performance/distributed-tracing.md index 677461e282895..f7192042d43da 100644 --- a/src/collections/_documentation/performance/distributed-tracing.md +++ b/src/collections/_documentation/performance/distributed-tracing.md @@ -12,7 +12,7 @@ Sentry's Performance features are currently in beta. For more details about acce level="warning" %} -Enabling tracing augments your existing error data by capturing interactions among your software systems. With tracing, Sentry tracks metrics about your software performance, like throughput and latency, as well as displays the impact of errors across multiple systems. Tracing makes Sentry a more complete monitoring solution, helping you both diagnose problems and measure your application's overall health more easily. Tracing in Sentry provides insights such as: +Enabling tracing augments your existing error data by capturing interactions among your software systems. With tracing, Sentry tracks your software performance, measuring things like throughput and latency, and can also display the impact of errors across multiple systems. Tracing makes Sentry a more complete monitoring solution, helping you both diagnose problems and measure your application's overall health more easily. Tracing in Sentry provides insights such as: - What occurred for a specific error event or issue - The conditions that cause bottlenecks or latency issues in your application @@ -20,7 +20,7 @@ Enabling tracing augments your existing error data by capturing interactions amo ## What is Tracing? -To begin, a note about what tracing is not: Tracing is not profiling. Though the goals of profiling and tracing overlap quite a bit, and though they are both tools which can be used to diagnose problems in your application, they differ both in terms of what they measure and how the data is recorded. +To begin, a note about what tracing is not: Tracing is not profiling. Though the goals of profiling and tracing overlap quite a bit, and though they can both be used to diagnose problems in your application, they differ in terms of what they measure and how the data is recorded. A [profiler](https://en.wikipedia.org/wiki/Profiling_(computer_programming)) may measure any number of aspects of an application's operation: the number of instructions executed, the amount of memory being used by various processes, the amount of time a given function call takes, and many more. The resulting profile is a statistical summary of these measurements. @@ -30,7 +30,7 @@ A [tracing tool](https://en.wikipedia.org/wiki/Tracing_(software)), on the other Applications typically consist of interconnected components, which are also called services. As an example, let's look at a modern web application, composed of the following components, separated by network boundaries: -- Front End (Single-Page Application) +- Frontend (Single-Page Application) - Backend (REST API) - Task Queue - Database Server @@ -38,7 +38,7 @@ Applications typically consist of interconnected components, which are also call Each of these components may be written in a different language on a different platform. Each can be instrumented individually using a Sentry SDK to capture error data or crash reports, but that instrumentation doesn't provide the full picture, as each piece is considered separately. Tracing allows you to tie all of the data together. -In our example web application, tracing means being able to follow a request from the front end to the backend and back, pulling in data from any background tasks or notification jobs that request creates. Not only does this allow you to correlate Sentry error reports, to see how an error in one service may have propagated to another, but it also allows you to gain stronger insights into which services may be having a negative impact on your application's overall performance. +In our example web application, tracing means being able to follow a request from the frontend to the backend and back, pulling in data from any background tasks or notification jobs that request creates. Not only does this allow you to correlate Sentry error reports, to see how an error in one service may have propagated to another, but it also allows you to gain stronger insights into which services may be having a negative impact on your application's overall performance. Before learning how to enable tracing in your application, it helps to understand a few key terms and how they relate to one another. @@ -48,7 +48,7 @@ A **trace** represents the record of the entire operation you want to measure or Each trace consists of one or more tree-like structures called **transactions**, the nodes of which are called **spans**. In most cases, each transaction represents a single instance of a service being called, and each span within that transaction represents that service performing a single unit of work, whether calling a function within that service or making a call to a different service. -Because a transaction has a tree structure, top-level spans can themselves be broken down into smaller spans, mirroring the way that one function may call a number of other, smaller functions; this is expressed using the parent-child metaphor, so that every span may be the **parent span** to multiple other spans. Further, since all trees must have a single root, one span always represents the transaction itself, and all other spans are descendants of that root span. +Because a transaction has a tree structure, top-level spans can themselves be broken down into smaller spans, mirroring the way that one function may call a number of other, smaller functions; this is expressed using the parent-child metaphor, so that every span may be the **parent span** to multiple other spans. Further, since all trees must have a single root, one span in a transaction always represents the transaction itself, with all other spans in the transaction descending from that root span. To make all of this more concrete, let's consider our example web app again. @@ -66,7 +66,7 @@ Let's say, in this simplified example, that when a user loads the app in their b - 3 requests to serve static files (the HTML, CSS, and JS) - 2 requests for JSON data - 1 requiring a call to the database - - 1 requiring a call to an external API and work to process the results before returning them to the front end + - 1 requiring a call to an external API and work to process the results before returning them to the frontend ^ - _Database Server_ - 1 request which requires 2 queries @@ -181,12 +181,14 @@ Transactions share most of their properties (start and end time, tags, and so fo Transactions also have one additional property not included in spans, called `transaction_name`, which is used in the UI to identify the transaction. Common examples of `transaction_name` values include endpoint paths (like `/store/checkout/` or `api/v2/users/<user_id>/`) for backend request transactions, task names (like `data.cleanup.delete_inactive_users`) for cron job transactions, and URLs (like `https://docs.sentry.io/performance/distributed-tracing/`) for page-load transactions. +_Note:_ Before the transaction is sent, the `tags` and `data` properties will get merged with data from the global scope. (Global scope data is set either in `Sentry.init()` - for things like `environment` and `release` - or by using `Sentry.configureScope()`, `Sentry.setTag()`, `Sentry.setUser()`, and `Sentry.setExtra()`. See the [Additional Data]({%- link _documentation/enriching-error-data/additional-data.md -%}) docs for more information.) + #### Spans The majority of the data in a transaction resides in the individual spans the transaction contains. Span data includes: - `parent_span_id`: ties the span to its parent span -- `op`: short string identifying the type of operation the span is measuring +- `op`: short string identifying the type or category of operation the span is measuring - `start_timestamp`: when the span was opened - `end_timestamp`: when the span was closed - `description`: longer description of the span's operation, often specific to that instance (optional) @@ -222,7 +224,7 @@ It's possible for a span to have equal start and end times, and therefore be rec #### Clock Skew -If you are collecting transactions from multiple machines, you will likely encounter **clock skew**, where timestamps in one transaction don't align with timestamps in another. For example, if your backend makes a database call, the backend transaction logically should start before the database transaction does. But if the system time on each machine (those hosting your backend and database, respectively) isn't synced to common standard, it's possible that won't be the case. It's also possible for the ordering to be correct, but for the two recorded timeframes to not line up in a way that accurately reflects what actually happened. To reduce this possibility, we recommend using Network Time Protocol (NTP) or your cloud provider's clock synchronization services. +If you are collecting transactions from multiple machines, you will likely encounter **clock skew**, where timestamps in one transaction don't align with timestamps in another. For example, if your backend makes a database call, the backend transaction logically should start before the database transaction does. But if the system time on each machine (those hosting your backend and database, respectively) isn't synced to a common standard, it's possible that won't be the case. It's also possible for the ordering to be correct, but for the two recorded timeframes to not line up in a way that accurately reflects what actually happened. To reduce this possibility, we recommend using Network Time Protocol (NTP) or your cloud provider's clock synchronization services. #### How Data is Sent @@ -238,7 +240,7 @@ When choosing a sampling rate, the goal is to not collect _too_ much data (given ### Consistency Within a Trace -For traces that involve multiple transactions, Sentry uses a "head-based" approach: a sampling decision is made in the originating service, and then that decision is passed to all subsequent services. To see how this works, let's return to our web app example above. Consider two users, A and B, who are both loading your app in their respective browsers. When A loads the app, the SDK pseudorandomly "decides" to collect a trace, whereas when B loads the app, the SDK "decides" not to. When each browser makes requests to your backend, it includes in those requests the "yes, please collect transactions" or the "no, not collecting transactions this time" decision in the headers. +For traces that involve multiple transactions, Sentry uses a "head-based" approach: a sampling decision is made in the originating service, and then that decision is passed to all subsequent services. To see how this works, let's return to our webapp example above. Consider two users, A and B, who are both loading your app in their respective browsers. When A loads the app, the SDK pseudorandomly "decides" to collect a trace, whereas when B loads the app, the SDK "decides" not to. When each browser makes requests to your backend, it includes in those requests the "yes, please collect transactions" or the "no, don't collect transactions this time" decision in the headers. When your backend processes the requests from A's browser, it sees the "yes" decision, collects transaction and span data, and sends it to Sentry. Further, it includes the "yes" decision in any requests it makes to subsequent services (like your database server), which similarly collect data, send it to Sentry, and pass the decision along to any services they call. Through this process, all of the relevant transactions in A's trace are collected and sent to Sentry. @@ -258,7 +260,7 @@ You can see a list of transaction events by clicking on the "Transactions" pre-b The results of either of the above queries are presented in a list view, where each entry represents a group of one or more transactions. Data about each group is displayed in table form, and comes in two flavors: value-based (such as transaction name), and aggregate (such as average duration). The choice of which kinds of data to display is configurable, and can be changed by clicking 'Edit Columns' at the top right of the table. Bear in mind that adding or removing any value-based columns may affect the way the results are grouped. -From this view, you can also filter the transactions list, both by restricting the time window and by adding attributes to the query. +This view also includes a timeseries graph, aggregating all results of the query, as well as a summary of the most common tags associated with those results (either via your Sentry instance's [global context]({%- link _documentation/enriching-error-data/additional-data.md -%}) or via each transaction's root span). From this view, you can also filter the transactions list, either by restricting the time window or by adding attributes to the query (or both!). _Note:_ Currently, only transaction data - the transaction name and any attributes the transaction inherits from its root span - is searchable. Data contained in spans other than the root span is not indexed and therefore cannot be searched. @@ -274,6 +276,7 @@ The following functions aggregate transaction durations: - average - various percentiles (by default, the pre-built Transactions query shows the 75th and 95th percentiles, but there are many other options, including a custom percentile) +- maximum One use case for tracking these statistics is to help you identify transactions that are slower than your organization's target SLAs. @@ -322,13 +325,13 @@ _Search by Trace ID_ You can search for all of the transactions in a given trace by expanding any of the span details and clicking on "Search by Trace". -_Note:_ On the Team plan, results will only be shown for one project at a time. Further, each transaction belongs to a specific project, and you will only be able to see transactions belonging to projects you have permission to view. Therefore, you may not see all transactions in a given trace in your results list. +_Note:_ On the [Team plan](https://sentry.io/pricing/), results will only be shown for one project at a time. Further, each transaction belongs to a specific project, and you will only be able to see transactions belonging to projects you have permission to view. Therefore, you may not see all transactions in a given trace in your results list. _Traversing to Child Transactions_ Some spans within a transaction may be the parent of another transaction. When you expand the span details for such spans, you'll see the "View Child" button, which, when clicked, will lead to the child transaction's details view. -_Note:_ Traversing between transactions in this way is only available on the Business plan. Further, each transaction belongs to a specific project, and you will only be able to see the "View Child" button if the child transaction belongs to a project you have permission to view. +_Note:_ Traversing between transactions in this way is only available on the [Business plan](https://sentry.io/pricing/). Further, each transaction belongs to a specific project, and you will only be able to see the "View Child" button if the child transaction belongs to a project you have permission to view. ## Setting Up Tracing From c6c19f7f9a0edcca82f3a50520798f3b1208ec51 Mon Sep 17 00:00:00 2001 From: Katie Byers Date: Tue, 12 May 2020 11:02:28 -0700 Subject: [PATCH 11/12] a few more minor tweaks --- .../_documentation/performance/distributed-tracing.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/src/collections/_documentation/performance/distributed-tracing.md b/src/collections/_documentation/performance/distributed-tracing.md index f7192042d43da..1dc49ef379ef7 100644 --- a/src/collections/_documentation/performance/distributed-tracing.md +++ b/src/collections/_documentation/performance/distributed-tracing.md @@ -91,7 +91,9 @@ Each transaction would be broken down into **spans** as follows: Let's pause here to make an important point: Some, though not all, of the browser transaction spans listed have a direct correspondence to backend transactions listed earlier. Specifically, each request _span_ in the browser transaction corresponds to a separate request _transaction_ in the backend. In this situation, when a span in one service gives rise to a transaction in a subsequent service, we call the original span a parent span to _both_ the transaction and its root span. -In our example, every transaction other than the initial browser page-load transaction is the child of a span in another service, which means that every root span other than the browser transaction root has a parent span (albeit in a different service). In a fully-instrumented system (one in which every service has tracing enabled) this pattern will always hold true. The only parentless span will be the root of the initial transaction; every other span will have a parent. Further, parents and children will always live in the same service, except in the case where the child span is the root of a child transaction, in which case the parent span will live in the calling service and the child transaction/child root span will live in the called service. (This is worth noting because it is the one way in which transactions are not perfect trees - their roots can (and mostly do) have parents.) +In our example, every transaction other than the initial browser page-load transaction is the child of a span in another service, which means that every root span other than the browser transaction root has a parent span (albeit in a different service). In a fully-instrumented system (one in which every service has tracing enabled) this pattern will always hold true. The only parentless span will be the root of the initial transaction; every other span will have a parent. Further, parents and children will always live in the same service, except in the case where the child span is the root of a child transaction, in which case the parent span will live in the calling service and the child transaction/child root span will live in the called service. + +Put another way, a fully-instrumented system creates a trace which is itself a connected tree - with each transaction a subtree - and in that tree, the boundaries between subtrees/transactions are precisely the boundaries between services. Now, for the sake of completeness, back to our spans: @@ -191,12 +193,12 @@ The majority of the data in a transaction resides in the individual spans the tr - `op`: short string identifying the type or category of operation the span is measuring - `start_timestamp`: when the span was opened - `end_timestamp`: when the span was closed -- `description`: longer description of the span's operation, often specific to that instance (optional) +- `description`: longer description of the span's operation, which uniquely identifies the span but is consistent across instances of the span (optional) - `status`: short code indicating operation's status (optional) - `tags`: key-value pairs holding additional data about the span (optional) - `data`: arbitrarily-structured additional data about the span (optional) -An example use of the `op` and `description` properties together is `op: sql.query` and `description: SELECT * FROM users WHERE last_active < DATE_SUB(CURRENT_DATE, INTERVAL 1 YEAR)`. The `status` property is often used to indicate the success or failure of the span's operation, or for a response code in the case of HTTP requests. Finally, `tags` and `data` allow you to attach further contextual information to the span, such as `function: middleware.auth.is_authenticated` for a function call or `request: {url: ..., headers: ... , body: ...}` for an HTTP request. +An example use of the `op` and `description` properties together is `op: sql.query` and `description: SELECT * FROM users WHERE last_active < %s`. The `status` property is often used to indicate the success or failure of the span's operation, or for a response code in the case of HTTP requests. Finally, `tags` and `data` allow you to attach further contextual information to the span, such as `function: middleware.auth.is_authenticated` for a function call or `request: {url: ..., headers: ... , body: ...}` for an HTTP request. ### Further Information @@ -228,7 +230,7 @@ If you are collecting transactions from multiple machines, you will likely encou #### How Data is Sent -Individual spans aren't sent to Sentry. Rather, those spans are attached to their containing transaction, and the transaction is sent as one unit. This means that no span data will be recorded by Sentry's servers until the transaction to which they belong is closed and dispatched. The converse is not true, however - though spans can't be sent without a transaction, transaction _are_ still valid, and will be sent, even if they only span they contain is their root span. +Individual spans aren't sent to Sentry; rather, the entire transaction is sent as one unit. This means that no span data will be recorded by Sentry's servers until the transaction to which they belong is closed and dispatched. The converse is not true, however - though spans can't be sent without a transaction, transactions _are_ still valid, and will be sent, even if the only span they contain is their root span. ## Data Sampling From 3945dcbcf00e33580cf640cf408b5eeb7e522223 Mon Sep 17 00:00:00 2001 From: Katie Byers Date: Wed, 13 May 2020 11:54:36 -0700 Subject: [PATCH 12/12] add diagrams and make final edits --- .../performance/anatomy-of-transaction.png | Bin 20397 -> 0 bytes .../span-parent-child-relationship.png | Bin 0 -> 42509 bytes ...ce-transactions-spans-concrete-example.png | Bin 0 -> 44707 bytes .../trace-transactions-spans-generic.png | Bin 0 -> 30408 bytes .../img/performance/tracing-diagram.png | Bin 23185 -> 0 bytes .../performance/distributed-tracing.md | 25 +++++++++++------- 6 files changed, 15 insertions(+), 10 deletions(-) delete mode 100644 src/_assets/img/performance/anatomy-of-transaction.png create mode 100644 src/_assets/img/performance/span-parent-child-relationship.png create mode 100644 src/_assets/img/performance/trace-transactions-spans-concrete-example.png create mode 100644 src/_assets/img/performance/trace-transactions-spans-generic.png delete mode 100644 src/_assets/img/performance/tracing-diagram.png diff --git a/src/_assets/img/performance/anatomy-of-transaction.png b/src/_assets/img/performance/anatomy-of-transaction.png deleted file mode 100644 index 9b4bce6dd5fb38ee196414b24c551393d7373657..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 20397 zcmc$^cU)6X^FOMhq9CFoARwS1qO?et5_%C3P-y}Zq)QFG9GWPgbfwo<^E7QUZ!1 z(i0#cEhsf~fdfeOp5W(szTbOa_qo5{_ufCQuN+MF%+Aiv% zX6fV7ZY6!}7{x7H-N){aHSbGWARPtF$TR}pj?Pf;W5=Xry`9Z0?5*6dnOoV|I>9+s z8=5(;*;>Lm3`8~WXgVueJ+r+BxLWA|TDlg1y@iA&hpfysX>Uo0fuohX*)?xR2PZd4 zZ#c&hUrFdYxf#ZB?TEzP9?tPssK=TQt|=m2t*(g*i1Awpiilnlmk;JYSt zNAND}&Rv+GFu$Ofq{tmfA)#yk{NsQ|bG5XVe59oE&sfkeIL9-0cV|f$%*)G5z)M&F z>1qQLl#q~s-4TKb3GqV|{BAx@?q=TnPHvq4;-F;ZX5nh<>~4#6x<=+`W{&i5hjT!Z z{+)uO^S{|Tx&5OiNMSH3;Lnjjw{fBm=fLO=fP{#H(q&$~j-?Z5ba30gD9 z9^BVe_Hm1EtoybJZl@+z_#4# z3r+TMdo%FiOL6`O`>srew^q=#xqRau)?FERprQeVBy^7Tk~;g3p+Eem;`<>?%i zmRJ_xoBZ@~n6C#Ub6`Jygz&KwjjDBXd}C((dTMh1IgsXT`}&1%`r`*)Ci+0bhqjPF z^y}d4k)a=-u$|jmzh`F_oNc2cU*!aNr(2jrxH?3d8v0upN5Vx+`ENWfDXg!=d|Fvv z%Sgt&4#`Q0#}uGzzTJH!D`xTFo(FM!Iyt^1;Z1?HY0wj0uXklwBOM?)u6S@@oX|hk zha1k$c;C}C__d|yb8}~WOn!NBqoS1M`WhG#n3a`QjxYc+Q_E2)Wquw>0p2M;f6Nx; zH5BI7Ke*@E-9gCCs0{K=_w!6lPbyDMD3!l!*feCI3V*?aQ4dpRA)QE&oin0 zD?SiN4St?cm|Gjz*ew9k{4>?}fy^6D`q*y8-g*DugR1fm$f;h#T$k{#wg zA3LT-tEMEc>pikGPP0GlDL?1(0v_GRpRK>#&t}%CkS&KP9954f{pKBft-|^8ldLl7 z1WI|`@aJ#|rJX~?a{>sy z@IQHQoWU3VK8tK7f>XDyQl9`U$ET82WFtPWCErTrX?b;;#Q$z(vNdH@ZK+Ke{&Y=;dp?dADBfqySP(c#XZEOH7fi4nn{u4oRIeq$rt$xq$LA zX~1ZmYM-gJr1ldjk8`N&_i-#&D}RPPGbySxqubo1LP;%dmbkLT0gqWDND`Rzsg|_Z*^V5dKxpWMbFbUXSYtRcAti0O$;Uq74FGluB+>R>bJX@ z$(1yus*GQYJM{V-i#i>Y;ORSDTx4MIHhP#VY0*IIq^SnX&O=nWp3Agk>jijj8$usLi92(0qUwq&2iS#i(s7$R#-w^;pC}eT|Yi>9@-sK5FZMRA<kEnl)*+(x~u~OOnR9nV;y0|tfGx6%?a@9yIm&9 z*V|-C+t`ei>krepJV3o$JpNHfOmgGDtX>Z4xw1=SZH zVBeXlZzj|0<3W^0w5i+KXQpS)lA;{AFQ0a#+0n&THA*DtWC$`iY~`M`j=fvSryQh%X|BNyzSbf+%~>yJ3de);dx@no9Bp#48z$GxMTI5<7;Dg~fK~ z80QYJ;5&KwGcFk|`=3x*R_v6)&Z6&J2M%}u&f30N*3Oz?otA`YFBMVa=F4s6!=aJ_ z09r$?f$U7mrI#qy1u*6=TUhxwna57CMX8@Xw=|iWfN=K@$bOybuhJ)rMEzEV)i+O)coVpH6f;6*oo%Z8 zZgEZ64W?P@ZeRPU?zLAwYlGp8lp!Op_4l|hM{byW6I#DP_s+QQLV^BB&h?dnOs_X2(^}ww_ zTb{kjM9qvx%a3j|&i(9RvAPnMooMMXVum4RGVI-91rBU+!foS_4oa8ZQ{S@Sxz0ts zLZqQ(%^=q?=r8W1Zpz&=4IoDIAW?mnuyTW}%kh7_f9oLn9Ih4*@5~54V27LL*|OhM z(DuCiRaKkai#%V4@A4Zrf!pgA45oZ@QG|k_1(LQdv>FNS^7IxhR$a0jAlYq@It<&%Ayjf?!B#v-Bi1X-+YC>BywS{q-x~V z7Afj3m9PTFmbZ-ft|K#PmEjJ+xuJq_1xFbnrO~{tw&L<4*c%i5cCH?V;dE?*5JmEf z;saD8ONz#z3R_4@)=*=D{_6K%R@N#rHQZ4wDdGSRDTsKLgQXhxs$ij-7a3$MY!=`{ z%Ci@w5~cLGV*nam4PnX>hxS9F;4sapu0fXLaGdd{TaE&AcE&^=gGd%TPqY3@c%Aj@ z%~`e;uGWwZsru34kkf?x3LWkraXJagAy4)1`#+g|OxkJfxjw-wV-ZHmtJDV!|DXhu zxq-6imH|=}vEweb0^31frQaF#oQ+h5NS~$Nj&i|umR}L~E9mn^ox`q&py<#^;~E3D z>Fa~m>_%BbE4S`PAR@{lxs~2byt`{G^N%{@X+i4?fnS!@UkKtVjk?`!m3kfTPH?Nh zDtqFI2xPkdvmitR`VK$9UnA6XK|Y}EblJE{qi2$0WDsHsPYK7R$&Xzm*;-`ToU^|$ zkTVpQITAe`vwR!hX;iJUp_dBz@hMx=u~9`x2w#^9zZtdJ#x$@jmBfB!}>qN<=(#j?%ex8RO1 z0^y&ArUkOZ{YyJCde*;-dR4Me!`rSL3LB`$+`QiXM*q}JIgIqF+L}K_tuoLiuym`7 z4Vzt7b+Kimcw*`C($KSa(2Cl41$+CZd2WVKD{&-GTVcj$v#3AUrJmgGyCFOP^wmTU zVXz*?d5pf7Ky`Rh$32gQ?QA%}8Ql_zvma59@G3P3jEGNUSquHTT>;g95SR{2h zMh5bjfrkr7sgmtBw!0-%M*BJZXXQfm8a>kRn7T0JW|Kl(miQe@ioR<0XheMjF$4q$>N5m2A*Y%6)duVNMdo7m4Z3Wbnq8p~{2uNE)>GBY2sCoYYn4i5O|;Zf z$MLe}iNUyJt!wO3jb!dv-$fHQI#~K9NiW$*nk)tLr|FwOCr`~8&DupuCCfQ`pSPlR z0iXzJ#wKIAiZof8iH0*{6IIRB$xNdzgnaTf;2%@wl2I>1sE-S7oQBvfd}8$nAur31 znh@#aNlRg_5xwPs=ek|bzP_SCyJ>?;H6M@f@yt)sF!H1V*~`O--1 zy0u29v1jb9ywsOJt)@ITt1b_c3>H~K#jns|>yhh;fvsuxOTYR*3n4RR1Gc^!+ipkG zbY*v-W+QhVT}GSHY_8T#modA5#!WX)0M!KM+M=8!{};xsIaVECDXHd~SaA;y%!ZjA-|J=~qcpCCCb;@+8he zgY86zl-zE5`niwW`*y2^zKPeYi9SlDx-Io)f+vxmX#jnRyOwIvdxgW-a% zO&yWjX7%WbA7#_9$C>V&^E~lIrfccy$BtvW!zYmC1D6!!kYzbF$ryz*$FUw^#5-rx zAP{fQSVr)phmwUd)jpp=`1w=Uvue)@rZlOaAvr$!ntDV`AJYPn$RXqUO0c%quAc-4 z?=aBAp$0TZ)7cmEJi*V|uK(*f37&xn{`a3O)ic-s2hVXhB>n&J`Tsu`61s8{HFpY{ zrlGfVzb*q-v#n~+Oc%N(*6p^XTlyC(x&cxzW3T@%tgI^OO0kHjXeNArPXi?|;n#*C7?wvGATl?PG z1;uuEe(6d1eWQzcR(wV0LF)^7vpmn{RXfDqvK`*jgMTbFO(tHsHYchu5mnx;B{mNY zms_WmpKo!@i7z59^5xc^${WI{pCJ2dkFI``oP8DKgmKzKwn0?SscXgQJ?6 z=%lh9v7bXummr%z0+-H3de^^V8R`#~(Mq`t6YQm@=N-Mxd@E<*?B%in{c#80DM#@U zOBYOa%L#Ul2PqaGT)YO&hdVGNIlwfnE0!Esu;;WjEe zl1-jW)32#58b0(8Sb0%t-qzJPKX2U|moq0x=U~u|vWWk4caYG_+ywMkFVfG~@4av$NY`8r=VXekk1@>zd(Hc zuyha1AvwM8TdU!f*Y0Kb07=e74cFYyo$1!Y{FY0p@CzP{;Lf~g}t&}HX+(7Cn3WYalr57FBIUP;nGEjuFRJ+{i6 z6pbCH0<47;B?HPSp{cGxFvZ6sww)_9!M&L#o6%I;I>r2Pj=@ml=5E6173vWne>m>R z-9hsPdT^3=aI8hokt`e9Cryj{?E6&x-CuWroeYXeW6FDY4On#d>Nw(qkv>acgB}?B zme?{*1*xfz87se@AgKmR=YCw-OfKYTEeuu8L=15MqW28nH+jcCu}QiZN`Svq??xmMqpP{aI|M?kMZXS$|K; z6X2QQTcDVA14Ky*Vk2EV0YKj%yvG?_3IN3;qq)m3{%(LL!D*jyS~>;ExHkErbo|~o zRn$0lu=`Efb0!x!Otguwx3u(V?p znuQ@7;6Dn%-8IcOCeVB$lQL6|!Qlr)CvkgohNg)EaTEH|7br{Phw+x%6c9ldD^|pa z7|ZY1+XT^3j?%n)RKBUx( zhbe|H5v|e%?EaLw@R{vhC}?DfXZvu0RXjjbh7*5zva)QRDUo!Oe{ji5BNWlvsPvWe z@D$|j)bg062gk7=%PY8cC(aSR?e3~eU5HWk&TH$sI2*=LC7?e>0ay6Ofup0_xQ~5} zvQxo3;6XlV!o4dIJZ_$$s72qVE2*10tq%Pgxxx>5M#H)1>)KOU$@5{yR#{u$$Z(xW z;c~L=bJo^7C=Hn9BGqZ_+F;#_-zABjCF=Tq5zt1nY9MSxE$Mo62D=3RA-?OMAyOmAmubPA4%M7`Z?sEZ*gHm_Bd zPFCiGTfhv%9#=A#UuIVu!u!tLAm#aco+z{1s7XR6c6I1uIZUMh99w$HQP1#|r;*Oi73`l` zW2Y^vk3_gvIJQ^zy>nkHeb&835&hLrxju?Wut0O~E9NFgBB(J-$gW7Kp~7n?{r0KG{P?Qg?Um+SAks+h^Iy}WK}6yPM| z&GuagT%YOFv*{jvWQzD{IO#~oIgir$@O|WVz{15oLy+xmKqjKjf1b9tpLe|PvEQ%l zF4Q+E=7c;=LK+>g_U;Z}NzxBAH?Zc2A178 zjn&J{84kO+`vS(Bof!^Vey)Y}DZk?$Qfx zWP{>E8XH`8;mQMHiyJ~kiLsG0@tIQ7TF^2&3ztgaz2c=tGS4=YgE^V9%uC9}K*_!4 zD^$j$*tdncwy51ocXfk>Ph z9B{u!c$-r&J5~KMdrq?bmcyG!r^WfgQX@q97qt2#FefE3= z5l4Yd5kV@Zyg(JtuLqqoi^8wAYW8-xwVssjT<@#iT^o}XSB4ptgI^wF42{Fig}>Br z0%8m_#@Y#*p4aN&d`B$ip{Q(kcDj2s%ex3MQ@4}QySB~GG{0Qde?4-Np2dSy7Xr)F z$S=)}dF^}eM)O_Cqh9auZjZqD{fM7HeeyKvtK)z}P-c~8-0j$Ksr@ncY?Rhcw9Q!E zLC3n~=^0#czZ3SmXj^YNPE(A|i#uE$HODh7dVSLNbX?q7DC5pWMwf^Urkty;Xxx1w z&hB5fH--07ieB;6`*4oGrObJox1pPd9ITDIy~i}-AuU5V7s>^XVmWc44U~4GRhOd$ z#h}ZCjuo;qf~_%VIG*#m ze}=aI9YeWjpI?G)(lgyECa6yFV`?*kU_aQbXg5OJD> zoeZT!fi0qL`s>O?f%X$KUry-un+^wv0}bLKfhDCAZgg{O$jtVK97L z&L+sZb2jj_lTNny6=~^b;rqI@|4jOnrCD9QkK{U<>Ekgt(0f@P|<(hB}bnlx2FZO^}mJc8*<2(I15Q3G8lBc0NhtS zO=hU65HtUc>{+$)BtcdtknB`v;rX0Uc_oN=I=ju?5+VwjiF-d(XUr)aH4zE%N_tHU z(smj#kUPwzV2L6QdT7t2+04bQJL1xy0oH}^qmpLZ`Sjfey-ug+wei9p!<>hUx z-^I3@bYB%A0Q|iD%Zud|QG6vn;p;Dor{lOdY{&9}Z!{%DSZn&ItH)miVi`?5q(DyG z1qFQGy5UUlD)L?xGQ86WY>=d|_eijJxp&UGiMaOF8l`T_<2VMmkUJ!%q1f|5& z3f6BF`)DYwe(Z)p90CB$NeBhQ?L&#pSBZ(}ZPi~@=fL+qJb^p7l6Tag3fdbv-N1-0?>o(29XTWoUf2)L$5xNL-*~#Svwu+<*m0kR z$-<6d1+up7D+3mnuZ=tZ&}w2ZD76YAnKU_(;Q8mVjQn2BRi#gj;>ufianwv>QRfYfG_8OE3Qt=atwzr&qlP0V}7xpizfpUd+WyaC4~#x`g|WBd9!nXFD#sgxs{dypYPn>^W>x zx4dOQjWu+)WNt4|cK--&D)(MuKNm3KM}O+08kHRE@O~s;1S;`rWDt828`h2}=Oz`2 zu!Yug**3{>MB2qqR9Rb1!}oC{P3Hnv^MYDROh0W-27u!jgOELdgWWJ;t8E@IKNM*TZ}+V{E9nWLdx$XdNqIl^t0H7 z_bxAdn3iy;UDi*^gSH>!IG(z*nt{;)CQxR`SVQW908HlF_iQ9Rh!s$5eE8NmM1kPkj6To0~LZFSQE~$oqf^6j`br?Sr#~PLUrJAF=^O^Ue$dULjw?^ zyX>^}(xSUKvN}@HjQ#>VYm22h1q;JVGY?5t{Rqi$BD#l04#|mgBUD}9`f|I>n!owO zl~>D`xLZCMaf8LIDFjsUAZyB+&(an92vk|Iip*qLgoPOcakY@{ON5GqQFejnLTseE z=xVm{agyU{U`Iq)R1uTZqx_)Nr#wI7$>K{y9;XgbhmgZ5%r=&OR%2WV#?+&Z@QXzF zJ-_V2*ERk6I}ORu5UszcTYzA~29BJ@YRbdXUUAJXGUrr<@92gXt{2&IwHXS7>PHq^ zhAbA(Xd}ksL5FSJ)`^g1!+zlxM=ecpWv1v;0P`oao3iJEJxRQ zV!NvP_&YUHUGeFC5K4Mkt%$+BHB`n}spkJqceWPUDMkAOISN-9XdfOMvs?*7rnCpq zeJ{#Ue=^Y+bvURNo;(a4nHX=$g(e^+GTz5l?JF51B%U`er~60C4bfdGsY7~D=J5jS z(4oHjdbk%e=LVhz``89ZZUazIF+L=frFDI#s;a6A*B|{Dp*@HConABMZIl}8d3-mv zy}hS6%V7CiDX3e$_$|H6XPgzQnO|~u%|r<;>GJ5tIzzx?{mVh9v>MZ>zG>yrK5FH2 zpd5xmN53Uidre40@R^Oqf^z3KXG=N8!gWKH3HftvSWCcUjzz__I>N@LFx zO!ZGJz}$b)K%f>q}^K`z#cP{uS*deHxUEBtju-vuPW?H@up8SNm;ZY(Zml zgwIyx#CDU(s*F7vKiWL^+m~ZAltXrW;+N~N>g7dBPOqE8yEm^ND@DNYkqRBgtMQ=+hUE2BeRZ)yjSdwb{>M3kbPHkXsN(0B{ z&EE62w};o7oMty0JpDIGFQ>bk=$bf}7mJrk^O=K$=%Nog^DiHP=h=V%C@Nq}m|p7} z@^=3$gho3r6q)X?nBf+00hGkU6%!q0ne(nA<>j;Ey%^-}cG@NnvnIuo$iy0=0fX;!0UL72 z6bR2P!}myrHW|V`0Y)0 z)J~jfRyUn$XGt_<4Ozsry*Rg|f$2gKx1 zTyjj@1JXsK4zZ}pqm91roAx%Z7+<9nmCfm_AvI88Q|qpGx9$#$?d_d3`})+El(eX8 zm<=}Xj4Dh?4Z7e@)L#{0@^$7-nnc$7$0sVfR&MRzC3#%NUQw+{BRncT*P-+C z{!k=E6Y%|kPhRci3t+)%DLcsoGB#wv{)k@HOloU~|3U#;$N&i7M=j7W?XbQefri+-d{F?*ILVoOwbc9jzV; za!Ppwf&ah$RsAc_QNc8boD7p!?f>dyEDOpEo6h;s9Zd(5Tqfm-5d~m(sg8s)xPx5A zOHq^G8RcyMY0{4~-OKD-`fXI;7CYAH2*t&Q6Xcg8+1vixhd9dI9m0^h+aFqYNt@1Y zDZjZ?v{w(k0eQ9vUXbX$+U&c`JhNv-a%7kJ^=VUOmBJLucEmipcJr&n5H*&c3E11d z+Ds%JTqSD=g0l7pxY3dOpwbUOSEptq;J$cP^rqJ~y> zqhaJd(-zqd766xA&m_6EBVkM6YUtzq5YKY~~4tk4SLD%l(N3@{*@R|(;7won* z#g!Jnd#D`Z=6$zU{N?9x)Z2YssdWmn1e(RbL_+6D_p)ELO0ajj)qLGfVz8EhzJg~$ zN9j!qZuBiHiIa}PV~VDo$LWKh@2%asU zn#_1G8x{a(gyM?FRcwW=@8_Oh6B!qu*xaA(*Az^)NoPB3;Pl~pV)DB=tv(dRRqP1i zz9?05dOgw4fiJu-h*eA#FttCDMHJJf%oQ1;hoDOS&p7#*=ZH2uHSMqR(iE13THm=I zVMN2;x3`==T+3g7)iU0I2S|B^#;ihj>s9e>4{EVFUz5uCj7Dkjf272%TJNhLK|u9l zvv29vtubwzAD$Iu9wzXnRGW_sCs*+Ao`venS+zW*5LCz*R?H!nZJ1u%$70B-@m2uK zmr73+P5rfxG*aq|cViEo0ino?gX=vXu48_C*t)v*J*6Lax9cmVckC}cwMd47#;FGn zQnQ6#mxSa5Tp51W`?UDTak`s*j2Ajbc~kS=0mIL~5Lqs5a)5oUpA{>mDm|7N2w&5L zD5QcBO3C!O8Qykc&3EUQDn*rhZ^DLh82bxz1LI!IYs=~#?IN?0#ETCd8*V!`sgOdd zKV(QbsXeAHai~qdDC_9eHS!}rL&~c?)VlWr7fbzJ4u;jpKZR~2Wj#?Yry+g3&ZfiG z`aSSV#p@)IsIprfpw1?FA3zdy^gX#=?Id#AQwY$f$W2ON!iQNt`T)>Pe{|wL_G@wP~7z^WZ>vzh!|_-uF%nIcDbfV_qk?G@~Yy% z{=Nb~juP6ASyfgeGo<>nLyeV;-H{_OMl>UVu%&{&+D%I6_LW~-ZphXl)%pA}7PI2= zxwYP@6*!o-3wHUjV!dNgM#)u63CX>mxg4&6XscActE`{GoCk%@xb=F zRJ|yWcHh8dvf*`j(7A{2Hk2ww4h%1n>TDUG&QJ;K-7ftcj+yxeYw;j26)$|mhk7fo z-G0qA(sHyVpLx>n&HT%4Mj+3uvK%VlMvpR`HX&^yp5C~-&3WoJZ%wM{L;Wj197~02 ze4(>#+@et?>dL3@4PF>heS@G1(z*YwiTIahswK~r#vecz;w-7m=ySdB8MEJ*R!*?K`hjsaqH}AcLB`k^f&l-6ydo(2n^xG>={Z2lsuE zzScIZ6T2K5_NP^L-L zd5Ng-Xv6Yzs;&~t=WdZzT&=kJOBgA&_GPy2*d%`yiYsYJl z6Jhpm{P&z1mJKJyiO?-?GCER(>8?h#$=FA^E_Ke#{y>>&(zEIU@SW*J%-z2L3i)KCr(u1|-@<07 zNN0KxA|Ofz%osHNW*30ht=PZ%>pFJb4UQA7UM8YW#NFnC*F65+7G;yyY3t@pwGK)& z{?dF^UhVMrVy*XX``|OyoWdc?>EAt7KGTgn=v@tzI8q@onI1a|lX1@+KcMe^SzG&8 z%U>0c6DGaC|DHH#%0pZX#^+QG7_daMziaOu-nW|yT}yMce#O~Xs;i4{*L2K!fqRzni_(Koc|0=!v4a_QUSayZPZhxTma;eQ<;9>_ctEl zi~~Wo=6cd{3Ji$0O$BzY3)fTYzc0**LZ)^ov~Nt>8#j6=t^S}))GvnQh3csH?pE(V zn2t>06Y7k}fg2V2v_HIur|zKS2JXT3owgeZ@~b~SW&Sl4Z`-{))I4Vxa3#{YWmEuV zQaDi_T5vy~<@EiqrDNn9?L;fl-WtXv;;leGJ-r02EXK>9>|Y@~Vj98`&xoSh2xfpC z?tKAD$$9iWi%+ctpVDybQ&_wMv?_Nh!%mbL1l&-oLi+P6YV-B`O96BGr?B=t2sU63 zk~Ng0t6!P6*+N^tVrSc9x;*9-(3fib)P~;>Q=zTCCm%C|_SzbbfwqFh z^8kM5GJ(ZGR$`~%b*BDOOJG@mk>bO|P}BsqX>y5=GT8X*ox|mi)M>TyOmJC|5!?&% zNq#BisGa0D`=VlrUsV-r<*6adOd)e$c(?V8#TKri?7;miVtP?L!z#Zijj+&Z{%PJ| zqNokKiMi$~Go%f5j>JhS6kDeL&mx;`n$GuQez|w{`HL{E8r3b2!h9qEy!uP7Y(~rR zFgUlOHgD94WJ1t6ZhymoQ*c?SE9!R9sWP))zeyM-44&yruzA&9IxM}DNW4Y3JzV=?uK)^DKbpj!kYXU$Ys?JDoBauWriLI~02Xj0 z`E1eoVhDD9G0oG~xyr6r`soT0)mLF(O-ktE%`PZ=kO|43c?#N{-px8bxN1(*Yg^!* zsBBZhre|^$-6&Eh3RCUn*NKQ!=ZBg>RF&gOmEGv_QadC#*d|@ zcLF?`=1S%T!x8x<+zG9IzKL^&_^rr}@;ddgvJ2SbLAENYTB4uNrOW|b;QfnU2`Wg8 z#2$HLWL3~k-+%e>5jBbCswRPB4iGf{$z;2@X3NS?h|4n~NNZOMgN+qx4m(=rGn07g z_wvlbMVGC1TRZ*E0@#774=KD%gAZbdc%xEZiNE-}irdI}V$T)0a2my>D769-)4E#we6#1^dA$zUXRQMs9+FLO zRSCsAs6two^LV^E=&e{13tof4=QAr^iD7@*C%z)m_tS<6QvRbNkgp{2k1g>Wlaist zR;?sfKXoL;P(Q92;LQwHDN@rPs;zkncGchJwWa~;+_jkCE@E}o0aDLP!}U|ga(we| z+BMt;^qGdUO>`o%XtE2G_=iy$sD8wl?UmLmQ?ZdN1kn191`lfnb9_f8l;sYIh0fYU z@nP)SKn15YaJ8}rRAGXpjTwq6JP<6yo0Y(R#Tqc&WPKCZuCoQba?J5gX!SL2(;6&z|V-? zTRHy*6+KBgArvRj^eZ!~zz*n_3%$EX(c9loI|qhR5T9OKv@!fPZOJ)baM>{jAqDmo z`91z)RPcCS!SF!|#^Awo}%s1*LOru9h zjMfz)w70N_u{hpm_LTgTLhEWF3A{p7aWu;gpgu1=*3rpwqegdUbJ2-E@A5JI3<4@$ zI+*BSt(K=<+kbc_(3_mlAeJvUy?faj&gQ~JE(e=|1?^o({+ZSvBdKv}HZ@UnF@S<(>)HIq-6;4o7`MW0wNVeUNwsEB> z4nj0)2Qi$}XT}-eHy&96{IO|M@Cuz{X6#S#odz!!4U;}W$ zSh=WZ>`1X}3f9bedyEz>bq<)u3`&63Vj3MhG-%qP90;^P=b}5xGU&Xx`Kv%>GMAoG zFMr^Jzm+Zz=zORvmM4chbad*s;!%cb^kjgf>C?szIjRb1EvQla8l`hZ*|oU;6twOn zui{28kk>emD#oCZ)P~U_kO2U8IJck}y5!*wdNNez#u}Hr{xh3YP)FF}C&LoQC-o9m zR?v%)zi<&Fya6%@dY=9#Px%XN{`p&b4V;zGpgMw;7&lV4Fx0*nN#vIC7!Ex}ZdCx; zFo$2RZW#9^-8}#ZZL&K9xAEkQ1xIqwD6O_dpWSHK=&;CxxpX9``1*Z{W>2QVc9$Gs zU&IUIskSabS1{jn^^{IdZ4g14jJ@bc{s(OWAD(XF8h*TJCc5kfh2clP*&Ss!tiCg0 zs|SNaf!i2ZvIUXDKdc|Kjqyqsk^l@vS1)E-S1>?XeI;&htQO(c744%EO+2UX*D)TK zlVq?Vvw8M4#DA0z++y7wv={iWx6+`UVvOK}xRi^I%`~w8iai*O%Tg63mj>eS4GS`u z;kD;Jvu4orB1C0PBpO*Gqu9t&Vl`D)RlTJBQ9GyRHPSzY%R=?`!CEJK{N88TEb>`> z?K~;NwVi9I2L??KvCL)G)BqO)GPD1l;}zSY<*=B7 z2NB&72r#GZ@GBB>jWQZQcSHXSwo(Qhn>!?~9uWN*Qk;Ez{a0SeKz6GYAU*6hmIoPF zA-Oi?W1!78+p8OI@MqdB>20RpCgT=l_A&}G4=U_S8}@_muM)F~+5sdB)9+po$V1NH z(W$j|BontSMwzVzT7DVK;de$dbgA7XbCv+lBR0Ok{C-+Q^fD@J$UsHU+R48!I5xN7 zE)#T&na(Bb{`0|Pt9qP-7yIzOhSDn4q=cj<&=>L<-bgk=BSC4^m(sMpkvyS0WMSyB ztK@Wn_#5JNgd;ye8Smn0z~miT6i9lqK3w2+1)DnBszV-T@eI&b&I}A8 z_J#{&vl>Y4&^a*Kw-5$6sjWsmMOf(^{Kv{l&Nk!$NhBE3|PR1Ftkri^Uk$xhPSVA~X0g3>-?NuYKay3~~8 z1)Rb*rf%bQVS6_rolq;TZZVPtDZdQ(h^|5h+R%DRj3FlM39N*(8q}z+#_P#r9(Pki z&CtjCHELGbir9a1;y85dPaU{w)x5J~Gm`4z%s+~L% zk%p`9qk^lEJptMaUm^50!nItpGntDonB$?6u%ZN@7L)n7*eKCjJdIJ_a{jDa&MC(p zTDanp-hix4>!uJaK)NV*-pRB^*BWsgu9wAlQ2t$?8qEFce~SV@wvCzIz4S$w)-Jqy zwTN&+R#jC`V_~a(us#V7%f79Jk&T&kd>T*5l|C)39%9E;UciIb zs+H45{8h1l7*c~lxis%?p>1Hs+>HhLq%G#Ih=2{HW)$z>#h)Vfpu;;(`J=Kht0v)v z_uQEGtEFG`yX}wQhQ)f_EYF;xQS@B*5bu_^pvzc|u+EadMwbA;6c$evbPsvo(0`~W zTy*ia3;(z9MYwqpJB;wEYh$4FC+Un~HZ;1}~iL4rh;9RK-7% zJ>WGEvTv=b6f`rglFKkx1Zqeji`q5|VNQ~;49iJIqYCIgfw`4NB|Fs<+P)o8WpJJ9 z=Pl6gAHmz(+tps`x%U^TRq9u1Rx*^9e6=JKs|)MCXTqvGpiSZ}&XYn$36t3E*zEAw zhvoA#8=7`_IuSc9>0#Kej%avd47^rZ`yU^6LWJfU$&Z8;SkHOAh%dXn6C7s*nlmSW z$VYb$6gXW3NMXAuIs++Q=u!czeiwo*1R>^a!iE)XlhZc(Yoi{7ul+5ThMI6xu}x)? zYEDvja%U(-yn=`AAgQj{DETSt_O;YgXa#3hUN47}fSO*-$I)wn2x@TpAD3YUYKob6 z^h^N1^-+mO4=V|aRu&jEZo{1Qh56g6**~ZU<_Rwk+&c&3%3^?XPu*u+bkmO|Ttiwf{0DcXMd{z`!`k8OcS#Heb zT0@k48m|y}OG1k7f=U`9*s|AYU2r^qX9U;9??vV#s&F9nN{wdLos;4L*$kv!+iU1M z*c;F-gPM}=mgWCr$xUisF72$lxO%I0aG1suC)JB^)<_? z;Q8}ru3s!3e(=g69gFn1#oP0YZmL}fI=g0GgZVvBTk?J5q*t$}tEu0f78m2*7ng9$ zH4a$!0n6Iy7jLKR>Jq;18p4y((KT0H%j)I%E``&+yBUB!k)7v$0i_-ZMjIWtUE#f}%zyUd@|23;JM5yM zqQ)$DUflkcdtP;h0^o5TaKi*t#cEeAGe7%%5!jTpMbE3|9shn0G=FyQ*n;EJYavr@ zAW3WpWtI~(NBOMr9)IIKpNrM9APuF`dw;K0s49PN@otCwpM&y$^5=T(a6T?u{{Q!X zRkBEN)PmS%3S*-16URkC!^^ecZSE>$T6f_)=n|SCv0( z{a)|){7Ch=R`IwW-_P5FS0Bc@z26pXeKBB9eD#-m1)dStcW+qy^Y{JAe-CxR_8L@$ z->Z7P_Iq^Ip}7@)*+4bGn)>hDbtgGWHYRzn;>DTz80jXhalO)K~XF;WtRVF1xiWJqi`dQl6S2!lIutzy;l zx4U0UuI|2P51C8u?Y#F^Tz(fnc**t81>f)Qo>a~K_V#YRyLWr;@t2ydUu_PKG_kIG zt5UM7|Id3}_vGAUzh4iso-UdE?$5;6-`?B@PIJGu1Fmu|sr@7#|NEBb>ONq5v(a{X zeBIquPd}ahF2ivaxRBO0pEbDs-phYcPh$_hTD`rt;(z1S)Bg_N?gi$pNf&=Vm%p*& zPx0A>6&L!~hw`zkULSYjo>$uHi@si0tT%eii@lES~E_VMtZNj@J6EAFK_%dUQlSlX`uP0F~Rn6={S2$j}opWE; zy1+fOUqGu$y7k_JEG}t9*%PN%ywK&^`+4R99}o6Jsu|qQ`5#ofq!qpSze`0ZR5gbO zJ>ZBedpzx>R<{fPSB}VSmG|A&wk~R~YDu`4wh&4kx1GWn8FjCv^0Y&KLA>5sg%r(7 z)ntj&j`<%>v>PY(PiTGKc*mk7^93*4^Z(j!1UE%|+A;e|&_(X->i8<*qZ1~$K2xH|G%nxsQ}yRL{8g5dv`o@%W*Xn(NoqaT5EldtUUe zh|kh3mu_g4Bt3i`E19M!b#>=^`Q@Sgp33X|(p~X4$>i<%mh_X(m;c`2Txz%C02*f>AqX2@xf4D<)7XLZ( zh#$X0#IG=Ga{TAJ+r!NG&r7leD)`SEA;!u0Ps!^oGN<74wcw{EE#A;t>;xj?tDkNQjOn(I~BCie6oQx7Hc?H)`|NICjX+y6Nh9t+n*Z zE-zczY*$*`t9D!-reac6E#Ln9_o<)!)Yr={D^^u94ab`o7NcVtcX~FH%^I{-?qmO= z#_PD%{^`xCI&=7*nqL!wXm_W-AdwfYr=UkXn3JDAof4Er9ZUV9wkfmk-|r;d)BbK= zRLOMrD#^b{V#uT;&pBM{W}!+33sDZ3_z556hXyOQcd_d#=w)W++E{pf z6GIL3jOcng*XhhJOdPF`)z2!B1S^E{@$->W%ZsArx5c9|Ir*r-#vM%b(y(D2zi?1X zUtjyqYWogN&{B$5V71$1O_aVnyMaKYmX|>eQa2TzYw1(V8;Tv}-ZOT|GSkUc&6}+^ z-8#JNv9e+nu18y5S=RkKA;E)3Slg?eurUKr)Fkif&ys!YV9m=VtI44no5SMC6IqGs z++z9N@r3rHC5-f59Rgt}d$9RC6t_Q1pPR_%j$It)auRl)v6r*jwu>((XEaILeK?vM zC;q8aZ@YP8f9mJlSm~LD`JG+b#kElb&(yWR65`_rsEw7Ozy_2{M;H6ie6$>j>Ma;F zBu?-v&FEa`%viKsxzS-!5r2ZH*Bxp;Qc~UWZi&xka^p*2`2qRG_iVeurR~)tv@2s8 zRx#1>9V3M*<6?15O4maezcY;Oi$qnZ7zXSw``z?LOmdoi6x~j1qCt2FH4rUy|7elFHre3%&-N$vmz+&^S1Q1u7nqq_N-n2tmaiVqRiUe}A&;M2}hPh^_?n`un$M&D4+kT81wv|T7 zqn6f=_`OK)t~M_}+F9U?Z)Lxca{f5@GzDj2tM?fMqV zR?YLuv}fH?y2)D|T(TT=z^uk+lYd(dY2 zBQ5>$!V-`83n>p_2Ui3274%r2Zn`TuH(^WF|IunpLlr-h=Zr(s6B&)cLSSL*@twD5 zYGEbA5$BN~+>pryD(`}7v4h?-a=3;`*0t7?{x1nt0Xm$`4;o@rHX*UEK8lu4t*EHD zKggE((*5Hz>6I%G_|M(@$HT`WhiF7QiT0N@2uB0&ygo}zvArvD4c;hCd^2=X%TY^R z-DztljEg2NLFfGK5?!rhYf6LMDwGbmHTHJ-PFzZ)R#5ebl;tf~g$eNo$+=aBl|~On zS#q;z2giG_#Tf!vY+eb-O--%O+U=jtw{MyoZ-x+V6eSD(6ntN|uivZ8^S8oCg*jJ- zf*34SZ8{ps@H<(m7oEja=^>nKbog-PdJvhez9vq2Ul83bnFI)=uC6}taqjR{Pkse< z1kQ2j{!#IgKt<-3akAe4t2Sx5>G8aV?jn8b@{EI+p7TWbxXLFRjP`ESiHF0&V&1GBO7q0R0 z(5F+AWip2A1$LyoF)0uk;I!L4c13lbVZP5#r1m|}q+{!qsau2w27$;^UHfiW^bVW} zLVJ$=QC4Pdc5ZHU+jA2wljN41kVpf!6L3Y_^11AVsL@NwH}Dv&YeOgQ=wzAS^R zf?7}E(V8q+%k7J%64vDzS4C6}oj9p))j_%6tZbJ|(=8=ur=qNqYm>brx|#+eg7$Op zjTfklI#}AggpRc*bl9^yIQ2yN{-&*pn2OzzozY&(5){az!1JAsdXUs!Fu(9M)Jb zc<5G2NO-!{bEgH!2jB$L&XBrIY`*@kzg-eJy8f(oF(fuRUJMU(d{%k%$*Rlik0<6D zGtCQB44IULz>z2p61|v?ra>iyOt*(XTB#il7JpHa zkT9o8ShOas*v;GaX6Dm^$~1?Lx)^?z;fQx%1Y{hBywcNe4V&9dzFGV&1g``LmpT{2 zcS}{y%3(93r2tZ%OI{jJ#Vi~>kvhh`IeF;^S*CEgETTVJ|(j-2RDrEvY_ zPRSzHb}(Glr0vK-P}(;+igpIpt7~hcNPxOeb{`>8&ohqpjCg{FeOqf?SfKRz%)@$B z%)s?^`qmijikhZDZ1F5ncBAuZS)9T+zIBpZ>K`18_nM9d@606)mAzf{JJo+Z-k3L# zcxQLb&f4ZwLANTm3e3!W47XZXNww5oA5$HCLM$b-7Wh z4ojtj;?au8fhBd-KZ%LWKNK0$()PJIapL!(4} zT}R)0?x5}sD>znpY!1{<C?QYZRq4J9o36fvB-`#%}Ez^X~L>6hOjFATXOz?HL8+J*XYu+ zhG_qourV42l)aoy)q^fwq&@V@!~v>jlnj!}Q;n5DMfwYbmjtvs#mrpt7>Akj9*kAp zg*U&Bo0pYvUx43fJ_RZS?;hhL@WpXEG};PoX{uL0ZFe7Mx^5Pkm2 zkNK%+PZiS-{@!UIs;+6E^G|H1G8c75X0F&_JrDSdiyOXXo#mbuLb}f9nusZ1UY0o} z4vmCI5|a67b2HUnEL-XAsTYvw>*6~}iY;rIpmp##Bz@};FRuV4;aQ5@Pys%uU(Z{q zgeC}z@(U-ugbY5%=3qAdV_7Bft#PtsBkMHntfRXHHy?$U>32&{@AZznbA^>mS|`Ki z9^crr?Be7V!~hU?N{y1QKpc2p3LWSlWULgH@oz$-KAlGVtT>|=VtO`t?%F4g)JXl& z=4WTMXg#yH*8l)w*S#{w*is|w*V|xaO7Y?arViEYwtuP$a?T8HQ#zj;X8uX7PfJHu zh#K1tHLOYon*v;Hv;&@5E^CL?g$x{gfpnt^)}a?-UtE8@F1JcTX8;x0=N=iVEqG?Y zj;dFV3L8-2@o74AI`@+vZQV9B7OGt}L-FRsUw8!hqQMrysYGYfwKz!?HC`Feiw$18 zMn^*v2UUH->XS3EFYZnqj*dE0-~UdhT~zZFV@i5i0ElGm#;@nvTJ%cI5N=-@HJa|lC1 z;*r1zWaFY9zeK5h64Kn7S6yoM*5~Yel-j^IaEcZgqxxs^u(m#p6!^C>uR6Qf<)Qey z#P#T6a^2+oM8O5k+rd)M)Zl??EXqkitIph!qCoMVkOaZ&ib$}L{Z#AJ&xpK@Lh$bH z@9u}Y3wpy9SnV_y1mfW}xAc#&x>*Z*HgJh6yqZWGY;b0Jx>LzB`fGG#F{J|heAk1{B;s?Kc3)x-kLU5hNz#qw>TU`0 z5}n7+PQ3hlf8s2fQ5PAk^Agh3pj{i!wLlcCxvm&Zu@BT$m7g<60-LPK&wYO=I z52W26lDJcmjCOW)v43^nZJflJA_aX^Aq<~=O5E!irCA7J-Ugo^x?h`8EM&(|Z6iW& z{V}iE(cnccz~1Zkz0iyQxI2V@u_vI9TgUEGDC5d$O>*?PMJd zc6?WDew6-55d z*5rmZ@RTn>Wbzs}oxz@_t6=vL<3xdUczQaNRp1u=_cjyWg?ylQwxj-<_@|R;Z#83o z>QNoZ+EkW#+=9STy?SQdOoZ`m1mdTd=7^yqt6AIgCM~zLVMzThh;Ngk4tLo;2m*QF z@of#?0cZM7mDW8X@M{&)3}Apd!hQ|PX=o*Z;E2_UWQ!ebMt%F3m~K#zc{}UIH5!E$ z85dE!=lJ}duM354%Wu+$W-Y+uicyx`WZtS2RtPf6^I~rZZ4|swR2p8jL0Nf~%>5zz zQ#)dyc{R2Sd4HIFI4SEC{;(KDSnhLDK-oJOO0m|;BQR5SoEf2#1ZgFLCKQ^TKn-ZO zEaQ&-DN4(OzpCH;1gMAe*RgozROH`7S^CuW@A1prBSKz0L?}2rJv@xl>cz@Ed=Tr} z#(0KRFgnKlAY#GS#ctE8Mhr!{mgHpW?rhD=CsBIoE4zu)1Qe1l!y|_K`~TKg?B|lf z_F5`NmzF66_=W!=k)esD26(QU4V%u-`{W4d0p!&E6tcz@79U5x6&6EoK6U1+B${R& zew?sGhJG;7d@7ivFL@JA{% z?e-H`h`vJ3c|{{rcYgP6KqWV^wA*gIqxP3@jXO^r4O6nh5+TC<@hnBO-p9(|V8uVQ z&+p~}n^Wtu5OH6)`m@vS#Aqo}OqE!t8CqATJx6~{VJUP9zSRD7F7>@USLV_&RDcH& zmc2SuB2p6HQ9Y)%1uGmbA^*|9( zZp2{XbRlL>_s(!cvv&RNBLT^23su25Kp3rFt%V-Qy^0YO#OI{+mEE;|ICQaD&HGn- z9R~5f357!6U(Ca_Lx4H@BIorSN*k`o;vp$nW0d17Nc5)3TEPROS8U7-Tp6*m@lanI z(c9Fg3Ty#XJ8sie=}FW4`Tm?=SfZ3GI;JOJh*aas9tlFZ4^5&=iu3~9>P04am z5kuz+gq6G8MPQym!7G7W?sm`#JnOrC;549@Ln@jlvCp|z>VzTl3*8Rsb38ODF1JIk zFMZ%Qtc|Il6!%{1G+7Dx6FPl{%y!i8!DjGv5_RyfZ&c=55;ARa!*0NAl3%%qp&Pq% zv8KegGF_@EuFgUVXb$Kkzy*M)w6^fN6?2nR=3Bo)6$2Oe=${W#wL*2_P&g#uh7DRO z|12B^_?)8jr4Nl0)Lm5Eg7(wJIVn6X6kkkfPs*;yLpSs=w*3qZ)1oV{Ajx}DBN*G{ zJ3;#L3d`^TwllHCZND9OPGAvl`6b{{xZ19oUyLUHBY^vL3HRMO&G^Y~$W7``N(IUVZm`v{fiYM<2#aZh61Z>;x(%~MQv47%R2+Q4q;INYn==}TGw7JQChDEi-? zRZGmrZk_%4J0U@P25ioZf*lYB3Bo&LP)u*hBmgH3RUfJM?!&Oy2KW=l!wNOB4;Mq} zCEWS3#$3r+s+opZl+$BbTdM%Ou1ccmTQ>PN|KK}bs4?+P>Ng7XB#-95jQh98lMvi7 z3tiKE_ieefq4%XtpA##Zmx=$tdjyOjElnJBhnJ7hK(C7ws&y>8usJ15jzKTs?76(% zC~*H0`TVi^2Ig9J?uQclyZt&oA+LzoAKZLSVY#ER9kbo4UA$54HrjmV?7@A#Q)l8~U;YEIJQhWbI3sselT#>ExX<>{t=Y3@70}64ry;Js3@~YH z$L2IVhm-o370W3w3Y<=N`sweJcKCHaYzPlJi1h1t5SuGkG1 z-OrzQ&)Tkr}_95d0y~@xV~Y%+VKWSAOlr z^2T6MNJo0Zz`Yu4e(SEvNzL)=jZV;sl2@QuDtk@D@VB95e&Y)uG_V2*icrGi@ALxu zaHAdg4OYL^^(D5;R`WI9_q@ccK4AI-?*wsrLcacRqw8O?#?%v5woqD_hJy0W@zHS_ zso3Cn_{9v_+|-ZLu0xRSCk*>yJRwKWlC<3qHCO^9ihQS#m4#m?I_XQ#ewSl`Xw zRt8S44492sCej2b@jVw_xLa5|ep9wEhBgj(*r0Y(V_MYoa@49-5k}FOq22_jUVH<_ z6}7AlU!}3m zr`SYfoEXFN1)is2|1$+Tud$wn`yHQ4a)j-ChHB54n`__dh;^kduO);HqWp#;v-~)?HR{l2hiodjS+ef2W9Gb@(I)dIF0rCw*&zXqt=sE`nMFnhi)zGh zNCZ}*W|Ol<1V^Ql&?#|Lk5ql%_~x1YSY3ah9WFC)DcjD!7KN(Foej^Q{3@xHnUkR} zR+^iWqjfO(Uay}dRLq?$@}L^$x->pm`>8yCG5mpbKTcERXyUas`OwW6@l7N8L3-$iSPnlD8x ze(FiNYyEnbG~)?oDcku5!el+92c_L$aXClh6;{X! zZgkqD*D)*lc**YYgJf2-%$q00PwBPwXn!Zy-R;ZKD_!TZlU*`T=eTR-9Pz{L&M#Qg z6AGfNiHD9i5Ob{9lv?47stQiOoH=)c1#rV6JO&(U9NOE4y#(wAIauZDF0xfoAk1%_ zC0S@tE6T{QObOpT)TprNVvDqEG@sIUVj_DvUzYqzeeg45cUDgNOZZmKMWY?(Qx6rM zZP$2hn$I8aj1Jbg8a*%SCSJz{*47@jaHxN>MPt?Wq^wuO1EcI)tJHzV9qyL1~NZ^zv-PVxpW6#sxkPXB!s*ONI*7>b15D zB7w6ZL#)q{$)slzecq~n8%3p_{Kr?H=IQRBeu3R4)Nzj!sNNCamvT!lMB^+Oi^;xT z!JE493@)Y8`P(beoUg z?0*EWWE0dO@>LmTo+|Ub1#Z}Qca1*B#Q@h@nTss$wsR4SakWQlTG3f=-Fx#a44y|j zv&OzJ|Kngm^?N=oQCXUR|?Vjbz<>no}3S_IttzMxg zAN0sxGX8}+=g;kjOL(jPt~XDLsK7CuoWO>@>ypD>AV-Ka9d6@x;s#He+JhUbtco&o zoc7{5S*gyR{;Os(WP0Z9vA$+yP<7y|mgF>i{D;a#;(+0h`utx>!t3{Y6UeF^SmNDh z99Z+2H(xl>8MyDXBoF`n&qU5V$7?P*UQ6>cjuYxXYI1G&!VO>zp+k+59Dij~%IOpM zfT=m@zfWAdg8Jp_Q49VaeoXx5-?D(E3vZ@MRopQ^o?B!Uwx)t`&l^+IbW$QlOn&q4 z7Fiz8%V4W<-MEQ_6Xzt8n%tz7x@v>$6Xoi0B65gm*e1*G4nLF&ZW5RN|1=XVQC7ge zv@{}muNSQ#;W5j3(se(`9jA13Jg(&ZxSTo98*d3_1LVm#I>LAuxD#3brZscJxRchV ziId8NPJ^guO}WD^O))n|oznGS37VQsrrNOb{HVRs%4KaA^g+QZ6~kmW(Un)n*ini7 z6CH+O4QO9G`ORdpW0S|L$ckJ{KK#=QmlIT2`97w2LQH`Rw7#jpCuZ5o;u~Z zcsxjda579*z2`o`t^&?~tmOT2PwC{zYU!0c259&B&J3mi*z=QTrR?17#O+$XQF;2Bi=!&n5h*Y&eb$hjsrr~CD@E-%SZ1Zz z#_u9>V;d?Q(cdB=x_{?KzR-aws$$eIu+2=8Id zv>f^8UM1qB4~9Cv4g69|?D5nYt6(o3v6kW`QhVRw#IiMou@l>Bh|J&_+&&Y~e`G%I<9S!U5x{~zMk+z0_(RY zvGcv-g0%qtQOZ}9TDler7b(nF#N-gGhVLZMMlnAP;!DE2C8FUdYFjq>T~|Hr(YB+ z&6lk9^~Ox(`_tr4_Xf~?;5mNYhp-xKTWEZh+hpjd_fI{@^URz;sQ?S`wek`4&+rTA z<<7Wr7#QRe>}nTe&cfXFITz8-@qAzOPlifiAi7rEQYv5C?~UWuYY&?DI@c-=7TgxT zrh6|2(2E>&`XYPdPpxyqp}Kp`FnJ#xP9C<|jO+UOpp>B*9j1yGa+&V|23Hzg#y3d* z`e&2)NwePmb|ugHPf%6CRI$}i>a~a(r$H{#D$)66S+}X%LW9I%gRGK?C0)Vz>q;el zqC9=%T<+0O?eSo)&Ez~xl%ONIbOJmcHE3j(UFhsA7vTq#``0C2xgs66b{U9LVk5gIEv z_Qs=lL%;}H@r@OkEn*j0%oZl}Xlj)}0_%IzEG7m9GSCrztc-yo97RQV z+yUMCd`(SG4gV^QBm5iKH}oU>zwxvA#fM4dZ(($VC&Z3kdfQF|od{gxGm^W~W3`75 z&^3n>CALX|AG|5Aufjw>%gNKpRsIn<{mU|f(kOlW=m%pA``c_TL*iS+PTdMBIl|5V z8BjTx!##Me0YsBLHfh=>Ei4F#icFhdGKfRg@fFNC(tZ)a=mWFQ7%7d!i1l!)58xM+dqd zUzd?+%NPt1$4|`v9Uci^olX3T{+aC^c*o};9_9Olpa5hO=?BYjlL1efBy;Gkr;8zaDC5rqGVnU#>=&zSmB`K$ZB_|eq zPiBF~&~?oo9qPNpJ`GxisA%ux6igr&Ld*^O$o_0)Qu)_f*X1cQC2}Gy=$;^s9ASWu zXMGrJ?e`j!(2e)&%m;|^Y{&7hygBNzLkA?X0GE%bluNiM^wxOC(t%+2`en!=9sUnL@skmxNpsSN!SXUHUU+-vOY(anv&ib%Ok0*nkzvqn(#|dlCrXQhy z!oav79;U=W&ZqJ_t&2VatMp~S^ufnEy^YQQSxRoBxh36-;A|tIp=ZL5@=>LbB|Oz<%LX_ zPFS)#=TGFnVXycfD9e9G`2-w-;>O|~TW&rK9$e){%ZktCfhdbYKlL2d&us_0PWdk2 zfsRI&Zm-n(__b1$)=(9Km?e~MNth-$>Dl-j;+ZLcl#O?zt}p`Z6)Bq)88m9mt6r!>NgpD#AoXb z=17#kNt;PA1&WjXQjo{$Uku$`%yMvGb^q$mHL=s-5H~bwCjboNq4SOEcb6TJoHDeMM+;WF_Pb z)8dv5OrqQ`cK&Z~TZ?B8^?~8VgDLMW%$~o$;MH(GqAi1UZ+H}np`2l%@P&j1ZgoU- zp`Mw`MpTINc)K;i7;7I@2#nj^z9RlhRcv;ubwJ%=u_KAZ^;rgSp*u|fW^l5ck5J7d z$FYOnqGyTW!rgnN+gSZ}5*e6DoR^t3ZJK-&3n>uB-Z^T(@?9@uYlE*ovUF)cAfR00 z=H?MQ1hO446^028S9ihc&Jg2NXC45BDPr$Spt-zE|Gx8{CRb`S zCoST}4Jf2U;DNfIoT#8K72k{rnS1GkycV?He*(zqCLS2;l8~ zVunm<_QZ4k^ZZ5=!*zO>!ckpY=aR8kgR={&2?VYz{Sb)b*}HFG?B++1B^_kkQ2rq} z^?lY2#H9zY(XC?CblL;yTPQaFO@@*gF07U<7q#&NH=Rv8EEy?9+C}tFs3lmQwt+AP zwVB?Pi{v3B$ch72_A3xQmk{sph3h5RmsgM3N->Ub{2&9F^V;l{?Rqc!*UAn!GQ=|& zHY468Lxj7M1vC=E5MJ3x;1pt$9DV?jp!=u=U^9Ia#8iIlSVLjqs+euU)h^d0$sZar%@khhN-?PKJjESh{jT?Dv5*k6vEIDhQdlu`NTHfWK+c>|>;eJ>nmqqIQel$PEB)fI<83TS?zLv+qZtCX2?G}q$TS1 zSvGZ)lP7Q3IWOeJ&#vmznYZ=RD{3AyN$V@?ReuTvI&E-uL|s)iF5PQ}gM(JG1-3b` zl1z_lK3T_c7g{!17v&;t%I@?E9Zhrj2;3W?&O zv)aMpfDlDU^x{cQ4ecje*}GZ|M)c)w+w;o}pJ*L|f+V!%BY(W8IB9sFI=ki@>a~GR z4i&FzTzk2~YKmdh-JnoY)l*fkuF_R4#P}wv$(q}1e!mm$6UQV&sO_ZQ@q|%ZmGG$rd` z*`#AGYjRFj5UX@*>g1d$FMMp#vQ9F#Uhr*xU|Lzd8WVelL8nBIHO~CA@zcC*H8e}d zW7M*MwHCcOD>6z0r+MWo(Z0+X7QB9gq%=p-7pbY_>@s98&yb(_F!Gez7p}|I6qD9# z70jh~H=o>l(qUQc{5Z7yviCD9jesTg@tq%%qXXZ01q*=O(5w&Z|7V#edXC$oNZG+^ z`1KLtqn4230`YF{x(i#`625#uaX!T}8kcnA2C_)`uAC7lK!Q!e#(m~m*F8qNzw=fp*{=|r&+SU4TBg_*opG~zygS(1Y-JlLbt;8r zHw);HZGyFdgwM-lHJVPK#=ZrSDuv4IojT3uY8^DkE$Q~vgyfgnr&r}%)+GM5oSVn* zF4HxdOiqQZed+&R^wvSAT>8?l{0BjYMT$ zS3-%K=RdW~Yt3!#FqdM>JO9MP{H;Qu-C+9~iw&&3r`RVqJ!f+|v~Gk?cl#agMfQJpT%kqFjH?9cDUO$CrPfK-5qvjp(jK;ZVfOkbDZuU1d-LueYr%hnG%}0PxBgb0mr6(POh#KwaJr!9Zoo0I+XY`L;l+1 zuZo>nHd@`T8dFl;(78LHW0&UDd8$lyEQSEG!lxGFdSqq>_b#c3!S99ZuRXvU5ubBw zkEWjrA1C*4Okf&a?5Ye39($oZf>NR>Me2RY#Fa6dJz0;ghQ9ys?o`4n#QHV-f!0j8 zugFR;JMFt3V?EH5%pFeqm=&}aj?*~Va|$K9Qd+~uT}ptGXDqC$%8Cz(YBuKX@9*E` zN4P{o;$?Qht%u_NjL_o_xpkYZkVoh?hq^LNaYq9TZZ~*Aj(hvYRXzPq=5(W`a2$a> z*|nexK(fYMxuPS5SENk$qF{F2vpI_>qXKZ091aYT4!@@?5MjDPKO%oM zh@+BEQhq;?z}^n*{+hcuh9lebfwIuU0|bHsZfPJsr`xl|v=`_-Y{Zk~YM1cVNUe8n zsb^$)7s`r|JbO=340yAG-`_5r)OtNMx!Z4!VF2OcaQsm0>hgJ}k|_O{zcOEi)KnI1 zc*N>5=wovhBJk?{MNMF>_84~kB~IXu8q`+|+LDOGF@U?*ufCgqCZhO|q{f(T4%8Sl z#ugSfo$h#IWVklw!~TEPxjE$=C%@Y*S&?i>MvHZ?t{V3(PX}`8EPIcMm%wIGJ&>^>26# zGo}bqQF_>+_l&*k771d^A={<{gCSEu9tQ8|++(Ns5NN*a0!o+U(jvHk1uYiPK!W@E zpodNxUbLK#t^Na6z~9z8mkcyd>*4N=b0li}{D@$0SiXA&JVE^3(CpJhs( zQ9-n7btM_k^xEhp9Ndt5Qd|5P+hEm@`yuUQW2AG)DNqT$Ce#;Q9FjcwghaPWbT5ui zCVol8{l;d3S9j)YG}6(cqtw1jy7F3w%?gj8eG1SiHzYss{~(hU04^*JSRBx^fFcLF znkvwC!L=9`HUDVL4A5Fuy@lnucwPs{myH+@-+?)xF$3e}{RCse-@u-+G?&tIYIctr zRi`VcF39-yjBIQ+ZwvveBxpiHJB`VwjnR2Y;dJ13y0>TtwmH7t>Hz=i@p4 zZEVT1qPFFU>s}`2%^^%&<@NeD?T5oWw9?r%cWJ-sK&;>wyE|^R{oY`q2lv`ROW*IWm^gzN*E+6F)kMWFBOCwW34C9q^*M^{)v! zAT#0yy%XCJNFg|Q)A4JAi*K(|>3@yKPFYo>7vJ1hUBt@E4)92+tnIRKGJw+wjeKS*OE?A&0>tfI z!b+?!j9rhY#67;vswUw4y5ir(cVc$EHr@n&(FwU2D&e$P4iEvFEbr+;ruvejy4U+| z3s-n2(EEK0BI1|>yym+NE}ePU-;U#!_C@u-+wH(#nOw5uJ-%p%3s||Tpm^#vt$eII zKVxQLMk1}3J~9_AnWVBx00%mK*JEg{57;g6Qt-Zu&w;t{t;rn7as=dxlGm%O;4b+| zSgl&jGy?5pP?ea~J-D4aHq@5~*NmN9k3c_NMkG~x%8Ilyq4Ef#*#ksX47+q}?#`q? z#r`4Lk3X&tLgh%SB3*6PMTnWaF$0EtVga8Bs8|a~9`X)fKh{_t<{Yz>5!;LNTS^zG z5%OMX8Yr1hRpR}#dy6_)3d~AG@8Xqew4;+=S#Zv}2i6KaZdc8Q{LU%^^+7giY&6TO zdj9AbA~{aDxrC=2@q|~DcDKH`D;gqi)!}X?Ef)B0t^gar=21WMqBYO%4GA{x@yMad z6aO_W6;rOBzW=7g=A7xM&*kE6;{(K1Jq@lU=a&o(2JX?x^0)prc9N@&s?lFugK!B% z?$uI7?^_z2(5ulq?%k;sDg8LQ1C+e8sXJW$+`KhA4wr@Fp#?^_tQcEb0-ylqfY%FN zwF+xAhYR0u=~BD!QWOZQ=;;7UB_I$d8qZ@%n#H^f0P@=4%E2Psk1jVUL5w{>4eXQ~ zaR9DWaeqIwbl5USChH|U-nqVfG2U~RnDd;`bq4Wj0aue{vfN;At8dDCaax>;wZ**0 zYXYwf3~=^qOOt|^t4#MJHc@sd*bq8!%v;Z1U9{5 zy2>6!>un->Jn&T6uVSRTW~TD__~w5^gK$BnYA>)-Gamg_UICAqRkNhQ!9bOOjBWY? za61lBnez%GKCRKkfS?JS!no?j|o?LoBK6BW(_h3_3Sw# zW+yMkmphU1p(R0GR_h{mshVh$)Jg4D9ZKz^wdoh_US>Bj0fRnTliGV->Xz&_Sy>Y` z)dj*VUfY%)F--L@{}6!FY*Q(TK}?TjZ;gND?wTP|4q1cnT^-Jjq&De{jwLR5%#pFW%yyF6B7nRmqsf>B%Po+&fyRT9kgC!Mfg zYT9_$U3*@$i`3s;ycqv^TL_Q{7Jf9rT%xR$$V%||c-U{qKe;bKk4yu=COFD`j9qOYYkgS6dithD+rfz1~Trkf59tfO$I36Z* zLL#}&F4aNpCt=0>IxE!@%A+b+>c|lv+w39?Rm&o!0#A zGry*gU@o|sNq;8MOP1^{>Uq)FtNn=l6Dofn$wFMIKSzxx%a8kd9ghRH7z-h{bRJ<- zZp5t$eHPtZoi>ctWUL&X>A)paSDp&l?nF1#RQ8FSSyZG_EqPZ+azOG+C2}C}(JxF9 z&uc1YbX!TWFDmXiOE+yH$>-Z5*XRO*hd!hinZNx<#D&D# za3tM9OY-pu-9P7pZR$2pS^5>)-I6R52psBMr}x}9pUgvi`L!;d(6yO(Ht%qC``pwp z`s4&>P!ocwtvD>rTD^nVWEor}ef}Ex`KzsV*%N}8uTk+|qv*_E~ssKhUp~xkj?v;vQ{{_)D?=3i~|2@y#s|Nrc;OxIsVpxBA23IT~R&f zi-$ckuXb!buc{RHPwyTR1jum2hlq;}c}y9i-xP9IQz9$9J*~OpgMV{6RC-j*kdY}~ zqz^GmhO#`9)v7;vv@WT}b$wXE{s(fZn@TD87iTULQ|jzqAUc@KpS9*Emm2%TKsweN zF;$~=G{2Q@;EkD?6uH8g@PSG$r23xFX7LisO^e-?(r@|OL6f@POLjH8{pn`)J@u{! z8$;b^B*wCtTiZ{1ht1!5cfB!dHBH{k-@cjrVOiY+S4}Y5Ec2jaiOGG(fQ={j1@G|U z*y{bF7b^VJ^{T_>Mnn-w&-;HjF}6`kbQ;@951YGm@f2G(_0Dk^)i^GEW4^c;NS0cj z#*-aD+-WZ|alt`#^AskQh57>-qA2PEL8}JnOREq1Q3|rD2DQ%H2S&N%((n5tvQycx zGhnr&`cl~Qj)1qwQ6_Jw8EPZPg}>6)JKw+mnVuHws5u{7Ew3O)yQ`DnQD%xg#p^>; zTj%K^RAF+hdX1sO;xk;tV3)qzv2}`@5L1oDb)iOLuG#Rn`juN6`&6|Ks2{ugYt679 zm6QFzw0&KLFb&58{E-;d-Y9rv!yu3SO)J$x`cST0c2;hDGh zlh`aI(#yyd$wn5J;8D6mnsRNXZo({i6jiLeQ-j7>*-@PRys0nmM8(gJ)vVaxKeLIN z)2dQMvX_v5KaEg;%cl0HXhx@t2i_>W%oX1t>$--8yxEb;e z?>l<>*bMR)8rs1HI%@7DviB)nN{t)e`1 z=+LEqKxiV||Hn!iSGJ!kj}dN-C(vt13(<8uUe!#~Km3heAncPhT4TKs_nBq5DEeub z=g1+Mejpu%52|2AbR)=RvYoi$Ku%)K^;`CjN^+80`pgx%4(JqCvsrYfGaQ=dy0NG* z+o7qmxiLMCx8`F{fGCxmq8z=x%chO!kcZ))nWaUa?CNA2`yZSq{w|ISt^;nZj>yl3Pn~zf(=ZF__^#zf9y7K51&kW?$$D|!ZFsJ@M zLK{KV51OzqlE*7s`3rLO;B0wny+yUBa;^#w(A&}R2~^sj2-G~~zZx76?@&lA-JiZm zJ^IE#RPFQEp7UQQBWKs>+Z4sfuYvs?!?w4|!-a5rlMD^%lL_} zgx17WQ?SUOEx5ERf}hiUFWA%7)2eh_c*-t5Y#t#lXBVO%bA=g0wlg4ZNRKe*C{S6x zadA-io~NsmiOs7Q?u;IpJNFAWFvBe4at}23)}p^ganLrXyQsU9f4wSRRkZJI;WGH+ z=IR`ZW?W29HUNvp1f4-Fo6J}Z2j-e^o>cd$JS+=0$bSIYK9fkaN~o-L7yD2YsX)lYKX)ZT z_heDo;7HFNkr;7RRY}ZmTJAQL5^f#sm9pb?*b<1Pipa{zit}132FTO6dv*MA?-9G( z!B|UJP+BH=0E~}x1(6s4fU8zy-8as8!~Zpd!F>)>7;(ik5%910DVKI%lDDcUtr&8vE3z z=LLoC(awu!6k#J}?R4p?Rd$9p^V7eTdlzUD`giRT;j z8l8$y2BLgVooS#D2V1zF46&k4&qiAIb1G>5AMV~eD5~xI6Kx|VM2P|dMxua9lzf#S zIfG;aNRvde#6}U2(14NzB(`J_$w?X&0m(EVIY`b8P0l^5x!>Q+{Q2s=dNWf)d8=+g z_c{CQz4lt4_4%y5j&64ftCRf9iTUs8`Pm3T@48BW)V_B}%`4V}&R`)MV)d`L6*&62ZH_b8HhP*u` zV4C?SPAz{P_Hb!e(%mn0Tovu79q5F*a*`0=bpVdP*)De(@2P}1PTb1IXftE=i}PN* zFXb|n&x`63kbX8{UXb|eho0aiZqKcKYWoh%15e%rhJd^}QHQ1TX+Y;yS`*@$Fx5`i z1LfS$I~wh|r)y@e>|53if6R&4kKYw#^H*hpXuKPr=*ek(+-eyE#Nhk1^sLM+ z5Nt2}9-e<(G%cm8emhnnHAR`{1DLR8TPQX9eO`yT#R7|Rg6P-$<<}+)kwE{oJ2ne7 z(oz>Io_SEjDeeJEVT&KT8SSVztpVN)4CzBqiQQjSe zhXwN5YtCb#_%S1nrWP$3p8MKvU@>m87|ormf!gT3asQqP>~*hw)dQwb{A)F&19B78FFoSZ;;2Nt*T?Z3ksjs9_v1MJ-l^^*n|;^q8*X! z-L-a?Gf0kF)#>|KfESU7p)(OiIoV$m74BlupYY$74Z1rr?G}{&>eypCZZC&zFhy}9 zqpqlAW$9|6sxOV6-uBnJz*d<33g^jO#;z2A%zxvNBdmc|Wz(@CFVhNaV$DINUK9*Dc)qg%jmktYL+H$E0Xu_ z7-^NzL!Fuv;HST8s6JYGGr$oO*w%D_kU?{@n@;%Lh)*RhMMf3&8!h>n^F70yvBO1` z>GQvq-yghw0EQHtB|IS7AK0Bf)5(b$8YjG%wKo=a*ef442Us3?Mt7I&yzlD`0X+HP zojLm02fBuFg2G*ND~(}d*wV7WVI1wzU*L?3>7wf5FG+Yj-C70$rm?Z#J*Y|51;03h z%r>Ui|9C=xn1K{X;AwuuU=}@E8Ob~>`4YLqSZ&l)K?3#r=OG)oydb^|pnxC{fYol& zB;D}k$Zy$9p9(V6fYtc0@HXZKbyBg+;*_qA5;Mr{%1{l2_$mMx{7Y3h9Bys@y7rV* z{*L(YpKhKv-rs@tPVHsPo(u2UBv&)Fc%l_j?R`Tl$J|Hu4_vA@_v0lU+}#I`Q7qVD zZ|aG}tI!-!4S6l(%YDFVB6`J1HR4FaT@lpQtfwlC2F{RE`&6VxWmwq3=n|2@a`26@`U=^e313H4$_h4qxp&7ycsSJ9)&=366vjX! z@qC4-F`xXLpUJa2W>d4Gp(ioE(%sFxL!EOj2l3bi=y_$5IhNi2B^(?-a_XVxtRZJ~ ztK7p%!6a`W~-)ZJzpCcZ(^_4Qbo1(v)-IZMLj9>3BLqFy( zPLNM`a9^>R~JkiUs zrx4@l87R)@emC|5mG!yQYQB`=-);zJ5Zn<+>De_l6M^(qPvo3H7Tw%JYk6!4`r>f% zo;Akz6~F#dWP8^_hef!ehH%xh_}O?$!~zJvi1tT;JC1Y`Q*yM^x*cI|4DEH_|EhM~ z$KO5@EGg*8c4=s6FT+cfhw-NN^Qe>NClHk+Wr#n6-LTQLV}9QERhN}HgT|AE-MT)# z`oDSXSBDn4^Ljimfya+{pM-L91cO#ry1S=m)djId3-q1ZqL1veTuS6w?39S47?SLC zQv3JEGmF)lFgdCjf^cnyk>*)#}MD0zE&dq%%I|MwCEZHSAr> zv3&h;GVA#QoPCqUnkS_O?x%vn!`D-X2T71jSJi@0)*kco^QE>UFVGLZVvS2ZhrjOL zV0wddd?>vtq>)Q$bL{lt^CB-Mcik>c;QeJtVW+*HQsF+mQSX{-cWRCXIb~liJHR&E zC2NQLqQ)K@EpOg2tHc_vnC?*i3HtUxR0g42U+#scr?~EKS=+p6jqP_u1fj+8P|1TC z3~ht#{NXkHhS+=`@kh#=-*vy6*_@zdDISiSrbm+22jQ&~7F6yjDybwMZGc)HC96pj zSV@{*Lah7aIz?z1$Q?QKz8)F*?s{U#J4FE)u@^X&EvCUmag&d5P0#X%2oGEs2HtDNes#Ma+q zlDpFtF0N*X5}Cs(y(XxCJM~N_`pfi-5$rx&(}?rSIGh}d=N{XL{Q?4i&*2TMeNmXl zC{lu^%7~rC$+TSZ=iBqng6)on_K&#x}Z_{_ld%mE<9wjMiAFZI-OHI!In9Ovb*hjL;5BQx+U)N`X= z1T|;w_4!1}32&qPgn{4b(prP6kd%qUm-M`zg01^7V+U)zgBspG#-?H9F~z6q1>Z4v z`~aSnArUHk?C@zRjq(hU1<>uI#MHvh71A-Wrzlljce%Lzo`ruijBG9zNUYu}(_fxj zViAs5+t;v#!ne1LrO4m2Mg#B8y*`e3m;}I}A;7v9Kkb$%4k@YP?qA5lAXoDp_zzcw zr7>ACp_j1#x+%37WCWS*%LB_t_U>{6khUurE&2S3B;|C1;qly_S)6?{bB+C?E*f3> z)bXbUBs@%}har_`v_dUHH-amA_wmlt7@rYu!H~cXbp%5nwzVqRgX7=~skf5-wcV*+ zE$VjuOvnkX$bC_)-^Dgpmf=((hv40AqzKxv~PF6+GEms57~XNyv+6& za#pM3lcY(}H!*3;WVCrW=XwPB5^h=ja_ZHr2v8kBHULnthx|BA1z~l$V8?kwXRlNF zdiwGZa5j+VS#4}xDsvgW9LLER4B$l@b!-Z&r*DWAb9-ig&3(49kM^CM83qv$9%Q%1 zs4RRVQ1%G&5rEr=FonNsJLy_Y&^=m|_ka-Wh%5wDT+5S;J?XzQT*)_r^tpkwR^9WA zhB+h+`&r17#6*Kvgs$K7w%_^@^99#o=r;mGbJB6bw-Ed}8zCiBHF7D%aSg2@p~`-g zztcrG6D~@65EO#pH!DlnRAiA>8Kg3K7{sE3LC5&{M#4BDW@#oHt-G#<(C8-``=(^NS`x`V2-E zTjW?|ZRra36>-*9gz@nV4->HwdTlO*FIxbi2$fhlHr3hGtb!c7xnQkZyr}?omHlg6 zHXS2dslYD31B4vw{D9@##6O0+OHnnFVN!4>C%+@wlg1_$WeRQldRE~X&z)bFOOzd% zV-cptGRN|=EQntasd%e91zB&3{%6v9;F7Tb{bx>~k}EPeTsBi^&S09GG4Rb!-Y7^1QSSkxf?#NEB%TijSl-~t@67q>NKTDnWF zVz-~fM0Qs%G}3jOZ!a$7ew{zR!JIKk#wKQILD=7wUQfPl`7$F9 zsnEIa`+9D40}yn;=yk%SRG=4bC+0Kv%jW1sKQ8(TeH2SeOOMC?h7SOFB{Nwwntr`m zIp2bDXIb{X$LY}xmB8vXeY3RVC7Y>re5tu8>BK5z{e0w9A)Ast<*;z^Di{pNS53&)3hE|<^TW}s#%MHTC zYx3E)O?!itkzU^)bSC$2ZcqGjjHJ3E@&I2G-0Valt7y*TT5)w(+9sfagq17{ipo4- zt=WG-TMGuzIL_62n&XE`3kwB89)&hmZaCpRN1F-kk0B!5$dW;awuI`oF+sjF$z@)% zhhIIngl>S}B=EwHtPkRXtP$hD*Lavi^ok$7D9D>ft-_)lev}#&D+b}i0FTZks zY9?GA0OvC&jvWsrH&-c|_}Qkz+vcb6q4xW8}pYKgr7y(%Hm;2kldwloFV9RkD*M3b9=a*!Hp*L|iV zui|wZzD+Tkc6~nRIomti-w+&Aag6H0jJ7MxdCsAw~McgWln9b(kT+@5Yd+*?Pz zhwzNkg|yy8*gZ7$cZ`ELj1wmo|xM7|x2YrEQ^SVA%SlS;!fhtB&hu2G^Ze-`sZ8V{!p@ zPhUs-Ql#o|`HX*&8rv`DD2Yt`6@mGvRoT9lMt96QU**z6e_zs_&@i~D7{YDl|v{PgsKmEqP z#Nz-QK;+{t&ahO1g>}ARcpIpqdfo*~dexraV@7u4$54U42BdY$84*QUVP1X*xLuJ= zT7`)a>AWHnc4>dnzBQPf@w0Ns@3c~+cqGlIO_GSTPSDJ$lxGi6?jQ*qBUN0~w()a%<9DMStt zZ1PqP>JQN+XZ=X)FelSV?4Ri&RQks%+Tu?L@D70}k?{B`x|$9!y1nvV%e*fhm!^)t z!`_8fm;3z&+Giv|UlmbM`MDL530wO~V;AQ|lqQr{e)UW~_F#78CsJ&v#JwD!l5$ln zlDegfY6(tRoTl#QZ!tB;bYBzoD87uMxcm02V$c#mQBZ|Ys-p&Jc)`nfvU$eOMTp}ShFpG7U%&nC0V9ju;P+eAUgT4!=nI$KH-eL3%$e_cZy2|TXq;fSyOv2a-ET_ zsa05`E2_dQ0%#dn|59JeAT#OseoxTZt@=&RbB3- z4^gZqd>(k>Xk^s0Co7aL>C5bKckEyZeV@lI9%t{6ov%8m%O-`MD1#RhpM4cNke|9+ zX#~mxt(K7UGf1nbp6i?aKZcKi5cZ3)r8+)-s5DZ~swRSiz@b@RpJpuP@U49KBs(d8 zDcT6Be?Zo2a|u9_@%ArwVLyUd@~vylt3%X7Cw~sE78BJ&p9=&UAK9Q-YXn4WL$Jd_ zsE?NQm#O;DC`+?{tV$+8-a`Sq>?SG^RycG2DcHUH&NVTKz-mXNN8deqD{9-g>ujmT z0MI?R8~c{2)&QA%Tyeg@6E~ilC`g0qqa@zEAnRNoeJI1+ev3Lxj|M;zfv5dRL-R%} zF~0l*7{ik!PfeF>c-h8{epO<#>RwPOacOb!^;S^S*11XF*EB*Hi}J*v{g_~5Uf(9n z*4MS$uXR(W0Y++RCF!r}XIEjkBf_Nl&eXDKT=LajtQt#U4oY>Xql3IFDC>EO`i*4Q zjA#)=(O1|P?0?UqxG<6hPfh`wdL7rTyVE2g+_l*{Scwa3CQ1D)ksY^IAV|&x0XjA~ zqX|F|4H(I!(dAYjvaAc}Q<95~{k%~s1Ng%@*vROWq`*~x-wO3Ne9NtB7B37hcDJ&} zDi4e`O55O7}kFOQLIE_plDF6tmjE#7TZmffgQlJyQ_dd6YwJRlx>M_9Vt@L#T z`}%s;@yZz&)DoB3v&~XTSE2NbB1hIZ zzL->&R|!d58A>Q;WmJ27`+P^lVN4F?!gX25XaF)*9ji~UMX|3eCm;!H1+O&A`hh_uzaQdwJRPhHW?TU|VN6~STq~UM<-HQ8Q#n(Q^+Spm=w(++ zw*6-Yuol5ce~ng`PebU;cR@%RlOSFcBpq~(LXg<74~!;5`ru*B`7NezsFUVTp_au>76l!vZ`Ofr z{0uy_fMTw`BDeJuGP3eQWEj2TvIA{n24sXEKLMoq-fauiJqoVt?T=Z)@v9Opt}h{D z2XQj!j11hXrb}ovh@b>S$-OKAuH6OoBRy$Uf8W~N`W880P=ro?b}%7bVjUhc7er3Q zt21)1$oo}Z+-;iYjwRsfsg*s-7iX>JQ$Z*CHvMWe-F;k6^wSpu&jbk6GwIJ*MR!35 zz|}gG9fXdQYor0v+ppEU?(<-ORJ&$ET96s%M=zG5I!>-mnvIHOtl}2`-qUi5$~!z> zN1;Og6A1~|s%(+dRy#1MdjC9CQaMY?sYq$THz*JAFtfjAuqJYISSm$pc3)Rl;#Pc) zbx+%9JT$2Ks{CYY8;BU$JbD1_XiGkLSo4C?IxZ8H+QQ$k>~SVcvkrsL&B| z-xyrtHe^ajcb5~mj?Lq2dvbAlJx$>E`^*3rLr&O!KNARb+ zdg!kfG=8Bu?dX`n%gpKPn6vQWoyLLD+E~muF1TWJ8tHEn1X5f8&xe|M-;}6hIB~F& zEG=cftH{WNYX_|g->%L<%!ksLw=4t7ay~=XW!R)fhav*Qd3*AG)gfgMG7qU&e6jQ` zTJxX}zH6K#f_@YkZU6rM_->%jYOF74D%K_$S!KXl%|^-*zb6B$;jxd_Z8}fQ$uLk* zQ9xV8!Nh{(1IJ_>C9SItd+nF|ww(qKepEY1|?vE|(cm^dZdT zEy{q~KbZN^{)=SaCc!8ZIvoE$o35rjEf5Px5jpOBRG!3mrzcf+iGpN?!s?<<$55w$ zJ;70(WxIoAw4<+RY$U5-6ShO(h5?KzB_MP>JeB_kADVlz=~&t9)ec ziQKW3>CS|ISVWhO($k>*^l>xu;Z&XKPL>{q@b)J8&SBgJ@G!>J&JG^d1&)Hvps|@5 zoUF}r;xgKFu5Y{=6s0z)QNE+?_>Kb-4t6)pwPw%v6gYgd3vL*vqnKz?%6ZivhPTh6Fl_^;$Hq+37J|l z>P>bWo)06blGGM)CaKQ_CFFZ_xPmA)m|v}CV{B=kU*ZzoegrcF;Iv>pOF%-`MYV@y zxSApSykMN`7g-7}+VZfRY@#;vRj1T9Vwd2qI3Cl`+HJScpd(|J%*ACTqx(F$?**=m z8KkyZ7!2T4$m7OmA4Sgh>MJbY*DT1D@aVkhKXn9x!P2j56_2|0YK5Q+L=*x!vP#lM?8Q_+RJ~;UG%`Pb7X6=AT-M?(YBJmVBxe{6Z0!D zu+q)oe`ZWvo^HL|^X@LA;9AV+h9kI-;}qD4fsmDuQaGOL3L~!vd$zmMR2k~V-wyab zFAm2&{4}%^HzfIzy&{5xx*@8%h}@;~IlP>Nb#9EQ?)4v+7|~nYgL6Cle}{)_wJYez zif#ho-)JJrxx`CVxeRFI#D-QcQao&9QVA-g!g1PUwX+s@sCoeJ ziUGZ*HY}(PnB+!r)bHu*Cd1#*jL{PpvgW(H}ud9bQht65DRR`j9_f*<(RAHSG(D zfvKL30f#&w${boBuU@kWS>sRp`x^|jD*%Tp|FrXAKag0U*_^p#TGpbtbBh8_h0r$( zX-c}Fh5geV;IFQp$>EQ00pCcfuQ57+bzx`X-r0flSIcw<*yFDK+s3{X6$K>)1bx&s zJ?5RmzrcG8v#6unZ7e3R0)C5dO2E+-s<%F~GonkY4LM@w9G%R#bBt$8IRls}1M0RQ zeb;gA0+{*0Z2ml30-XAOOJb%Yp@EDcPY-{3T0W5PcircuHa z?g@_?Eo-~LI}uE(HI29sjwK^cl3_L|Z5X)hm zn^K(zMo*1pfKk%838N!W1T*-T`FC$7us#T>qyN6RBV%UVmdGbc#@W|-P`>8F)KcjV zBFCU4?$l1d`?Vv8?=SR-_%RpT{kcV3bEYF16Zt;G@nfn7zEeXakr~gT9EkWU;f|k{ zNHkDQ1^u@5%SbFr!qaN{ti=*?H)g+WuxA}B*|AMb=gU+&c$S74$I%YfY0Z(L9M^YI z%Og_~#x~wgNAPfR2u9W9w!)X#m^-cU0cW)G-G@SK+dKh@Lo259%_>PSw1{n+x*y9? z73ZMyoj7b5R}`Wo5{VsGaW&zLRuuhJnJ(bsB~qm>7ONmfw0Y0HkDNvA42u#|?bRQi zI=n&9Zwh;rp^%cB(w{SEgrIC?(Wv*dbgcISD06T^Lr zLs&g-K15AZ5`AZ(7Yu_{%9cN0I1OiY%9RE;1tVjiz^VD{Nd8<<81`9Lwnx z)aIpAmTW(mDMW3)PCuw&?WKG?|F7gU*?N1Yos}&-RK^Tb$vV{Ehvff)buGjxH_5Pg zZY-s-ER-$HpBQ92yi>ROr4hqRznM7Y_Bz&4ru3Z}n^@I8t`y%CTEOR~6$9>Pv{!V=WHhhZL=eo^`v*~`%l6EE`SFMO>$Q!}?(e+?z-u=yA{1lbv7{Wk-MfpsP^;=YvrVZ3L z)4SX3JmJIP3e(*VbFwZyb!t2JCy}zEtO)O1HR#9&$DOa}rQp_lL${L$%MCfJ*iiDb zO1cUhW!DjU?_Hz?8Wi>IMij&nB)>ZfUQ=UOfqpQG$3SajP43uuaU}*lqiASt0dcLEI>H-FCnAKi|e5 ze%o4+=U4Kq%X8xQ^S9bm+8Sxt|CJqj@UnFL+~8T# ztRwA`zuFpk$%s!pmBF|W-&>b2s@$uPFGubWB31WS^mkDkng0FZ*r3<$hY!GQg)va! zyn#8IqjSNJhf!DQY;Td_+KAWG$^}sY)7g9?;P2$$^+s!vT7Q%&8pf-V7(&~%K%0L z+@X(BE1Ec&Y)E|wU=WX~s9Hr^zLV7wN2BFIV z>u9<5UjAWGDWp}IAJDXmMVzB_nOa#h$Il9Uzzx4xLRLCXF09rYgGT*guWEhxypBDj z#aJ}lP`1(xlIr+!7hnofmi&ea!Zmk{!EN)OAWfqhdXQ6Cq>F_Jv?J7&wgZ(kD^B8| zBwXwVR3rFF%;IT5s3o^plg*Gm|ECcqjQItKrWz$Lp?QWds@Rc~-SHURW4yKkVrxc1DHki9s}rvL_;!nz^Bhp;pl?dZo}I>U2&GCdKCx z{vMPm#9mAx)j^NW>gaUe7jT5+Vn&Ws+|0OiggX8)6#8a*G6cf>g3k4XLVDm8toau1 zEYyF$LsiUo%(F=3uZ(cforonBejze4lI^pB>iH`1q=DeCrDyLXFr@=CF)HCpj^c_~ zS*N&cxr7YDS3^1ksuk|iG(0&;%#_W~4Cy^M2cVMDHxfrz?T9xO&V+~1bDWA054nov zjgKaw&yE@Nmee|^#3opK!9g#mK$nr}go(xec`hdkC$b|y>{1MKJTW%X69r)(Xtufr zSr$0jv_4Ua(gf?4CRw+Xva@)p8?bC&C!nUaUzC~GjikwfLR!kdYOz@U2QtYTG_aHK z)G)Ucp&}w(bT`e~)|uZyPTUcow!;>s`5Y5Zmw$GsxkUOa&zl+vcG5f>Empvu$P z!fqS>PoUM1HfEhda{q^3A+a|3XolBk79hVk@%hwN8B zMZ>Ei6;h$<;!J}zQOcG&9385>xa9i$Wrl5bs_$u`dv`p6CuMJ2;EAyD28QRay1z>YW4v#>`RZF)MS(N0MPaw=80lg2M~kDO$P}n95L=&IRkwM* zE@()qKZOy`|3&R69c2uR#_UUm$9qDKe03>AgpQk=#{?80e!@>6OMr3` zdzRWi42y|$?tl40nrZ>`GBxqZb6gjQ{9=MRN~zl{K)DHDic7dg>8vhx&L-nAf8w0; z{=u<%N%nnGFei|Lh<6}^VCfDB*6eToEwHg_k}VB0kZ}pY!rOh|9jY(Q$=xwFaN1>H zF3IBOfd^Qp&o;*?x1gUtV7@@d18*7~fBOlmC?j!fu2EmBqmtN4&p1_wmgH0lFwpv} z$Eg)83o;|71DF0RJWNMJ@ES#U$BW{EGT_v^q=j@BzJLQgLy8JXu_v2zK_4cBWrU@w z_~Vy%El=f^L>Txsw!ai*j*^*O#QnUoqn{8l2P+5v{=<*Vi(g=Z^W#t)9UrV(dm$ph z396#}7-b5195)?(sf0yh4QM3IMHJE;*_~Y1WcPl(Ya3k#-PO!sI6A?dbm*@(%S6Ow z1xVA_$r~W2jnPOZ{8ueN-WLgiF5_bvgy3NpXu+Q4@;`Ob{sPZ2#nU?KIAjJ5nwe7j zBJeuk2KtSoJrsODi!QW*w2Fo8&Y${V)+?kngY)OFVwn@aN4L6Pm2}mDSA7)z_3N}7 z#Vu`~vhzPVO7m7T!*=~y!ekkhP3Fib_9n$E$H{v zlEVW}CoPsii}1ZY1N@PyA<$_WqbMs5&&V;lJU+2um zH&Bg6?Fhqi@8{f$i`vWJ$ji&?Hx`n!gg~BE%&TQ&FLn_#-n0p59e_xRF%i}*T3H-W z5M-33AjcBFt9tBqqINfGB)Mr%#46Sipdf|Q6$lilcv^_g`{bE#sG&Ckj7M=C?$xM! zgn=?@yfnR;B8f#j!cT@OUdBA65!%MVT_}nS~lmV@NA%?42&46E}B)EIb?5H!u*mr5u>R? zPQ*(wzCHlp5=o4N`gQgr!E|%9#T$X&pyd4^!5j`2*e<;iOHlK#(*v&!^Ae^WVkibD z%M8G`LMghq+w}K8JzKq9xa5Q7FH*^#eaQs_AfFgH8vqp;R1sFo;shYXFIV}XW|C#Q z@8lzLZ=1vXnC9la(!jiB+k|TUlHz?3?LVgS7Zuv>N!%kx{!63aO!JaQba9qY9Y`Xd zJDi}ZjGdE-9JWo$*8>PSGnZu89M*a(mO=q!z_6H9{=&^6cE*cb>Fp^m?T7*IQ{}ao zTam{NT?1ivrrem;!Flh1K+y+PG>OP*kTM!&OUsjl$iX0O^ z{e}{sZjgxN-aq|VCTLs(B>}}11v+ktaEXs|{KmoM8l%o@-@m%0j=;kJUb!dICG?1r zpxYmX%ufY6=J2?ZLFC5wG(|*`$l*3n9-q*`EplED;`057JJXY3ATzoE03-DTZTH6W zk2ebyK4vR-l1imOdvF7fpRuot4!4Io5>-$yLOFNC6gKazZJTgbOarqtOLzV7d{k4( z+rlw0K8R_!Ul&t{noBBrIi0b`y?-4RD90hg2D7Z4EH9*F=!ZO8EZ2?D zl3Qg*gvK&iyf~v9Cv7L4QD$>yi#5gZycTT<;-O+Y4}GlF7wsuZiLHBXIX;RJ(n!G zngB$*H3<)doeGuM=nXHIbTC&md(`suy0t<&pYIk2y#6B;zY^>J(sBIxsL8|H`%e{~ zO23|AYYjQdcI00S1YZ^3=!Ld={r2MtR97dIee;`Qo#J|%tZw~YCH*s|<)*?X{Bbq3T6FZ;?8xB^WR<2!7V|y5 zu~{S}IR91^7o|ChRADa$PnGh<%cpm*{>V6vnSDFMp+9fP8EN5-x%4Jqi#5D)@@|aM zs9`vowNdsm0~hlNq{QJo>M@!bl{>PUbVbUWb!*OtIInVaO_+7y1~HY`Bgt3Fm+$Db z#@-!87D(6575be6WQN}Dtu{Eg?HlE5Nc3|41DDU4n~C3FW-wKbN*jg#sA-r z+mP`4|CVzE|F@68yE{T9l}Qhg0VEaaR73}3GU-f7*1x0?;@=Y;$P+RGDYK3u<0n`U zTI=3J$KaD9cV``W;u}ouV+?ts9UdP^IzzH4si22+Q}oJoma?5O>*AT>A-6VPyW0GF zs=~W_^X~(9I5KsWtm1^&SMzYYl^92nVm7Kd8Hbql-NM?|FK?~f_eMW29Zj-0w0gUJ z-^3+5NBn!1y}#q6|DyUo8@_;p4!nDkOU?&1F4*1u-Q{MeZm1Q|**X8$epR?n3 zOIbW#?QY=P!oxVU*hs{-NO+#QN_rw3kzY1h$4IS`o(?_#-k-%yiJP)z{0G^*q?hYb z>oqpYo6!RwA4BWqf$|FlKq1sn(i)k` zVTP(6bNTiCpH-ZeTC4Z^+z2Pls+RIbu&Oq)NwgFaYf1aRzJzPoN!}arehx>s4!^YA zMbRXioG-TWZgc}NDW8K#qNR5~eYl;j9QyPByEZiQ%dXem%fGKv$PK)2n!K#-&57L{ zV(Ep~WkXX-{a+M|`;HG~uFG7_{0=T97Fhm}ZhkLD5+xc&O>{dL*;^m2E(}NShaQZf zmRX3ksA^Z2Xf95euPZ>m$rDEXN4BGhd-3ts3vCiTp8uXH^gdCtmt#{*u>y|SatdI_9+x27N^95xn(CDxbj z1Bw`dB!8p+*ZB4bd%NXN4-8Kvz`WYB-(+(tL+tG$3aI358Zrc;IXh?kY~US;z1L3qm0evIxLVNv z505+je*u#Jk1qh*$3lL@1m0+rj%Bv#GvD#Tk9w-!>mm(9nlcv66H08Y&t`eClpng_ zKG%Rv%-&>B54TnI^mN^GNpW}HU#!;QAnc73@ho1VQRpu%)w`N`2i&6c@O|%xl!dtB zZArteJmP^$)XSMfczbgFX4Ek8@1m`=>VDPca9#v@_`r0-)4ft%m~_ir&VW_4*SCsp zgcZpM5euwUACby7Ly{LqzJyFmEJ=~(=R)+aJL{ohRhR@ZiXutlYNt`-?S+x8ro??l zR3HAtU!v{O!Bb2HUd&t6%8F^5n2#nG__^`u zm06H>Szm*hb!h##D{#@jI$3UPaZsUv%BK0+1lnhB~! zE*|#F$i0}@o&%@70rN|_kV_*FE^nkZp@I1$kv?V#yC9`sC*gPACpg=!9sMX5nTF(I zF}ZEfgc?6mC9a^qYoIbRGpAf+!oYaBk((!L4@Xg_`hCgvp6C8T`Mh}>+R;s zQz*hWM)J9L?wPeWx^$&q76e(O_)|!zX_2ADla?$Q#_q`O7qK~Xgx}IQ;^_&u-!WQiQCTuP2?mFA@=48%Ui%_?n!EkI78at)k<;**IKiEyY zZFI3_tX7!X>Kro4FIt^62_9CMtjo3@)YdT(8qJd>^LGp3cVW4gkVd|_IGmt#Nz-)5 z*Fqqv$XWC53+9piwGYwOdr9aeEmkx~PG*+%-jBECbGMA6%Wt>$Z{Rd{F~@Yp2)$LR zg3SY&B-6uEcA;C}97OSH`nd#bglvfOszk-!bYi(DAtJ2RSH{(1wPVowS(QyR?)>D$ zgwmy&;w9HR`|g;h&F~#7it4-bMwgf_H-0{{Uz#OsWjr9MmpYyE`|BO`JU)wNr7v(e z5P?`#6%;v}*4&eFi6?<`iiR%oPtW9VEJecIxXQfkt_aQYbZZCm&SaRxRHZSzguU3o z@GC~j#poRam3mb|!w(E$rA=@4{&bUO*Qc~kc*g2&S7SR|gu`ATANCIJ=TThq{;Sh3 zyFM2Co5)LYNX*YursXF~6PHFDqguk98t?RzAq`_wNG{ud9KGL7jt>4b}v?T=9 z=W#ep%T>5lhB7Z%Nks11@Q?0ehX$JFsvp(x&snSp;4Duz_`Bv8%XKo2d@R_h02@)= z$f%xl^l;vr`P;rl_@qYLbCdsAu8pork)ilD{_Q^VR#NVKOX*59ce!~}O;dHHKq-5t z72w@l=ip;TwwD>Vl5Bns8@ey;&qO7UHTc~$V0LbKP5%Mi8uHCRlvkM+ITh*geu=tU zOPjb>9zhqe)Jsc2OSQ;^@p)h|ce813er#LcVll0wUwP5p>vHJm4$6h~rcaQ69Ij4| zFv8RS^~+T8e#7-aPj{Q~=E@6g{Fu&KiVr6FMFn;}Gg8->^;lQ?CJJ&nier2QvmVTt${2O z*jKA{xBbD6?(+#o%)FXiw_2)_G^>{5F)3dQ@2MOXF0mZpON&3kca<@oX|@Lg>$SGX zr`h!yM#qzobhDo~Reqm!aA5{!J8x+)&O=y4n7`u>oEBJ=-c2PC=5t&6*zmQ@mAA1f z4tp0#F6f|O&FEu*5`(6WNss5+D-U~5;!^8p6K=<;9Z1# zqMRp9f#xg}Hf4aO_-p7yc^7OL$~|tbdW9NdS76iWid8!3)#xPBT@fw7>4mXseChyy zm9pOft_F=CJ~vVT-t2TXU47}O?4}L9p;v8(X@`)Au%p-XFo~xm&c3GSnTe67tIMms z_XKSNf1gW~a2Y2u#^@UI)6qZmLkIX(TX#DBW4X4K!exntks)MrJwc9{ z({p(MjQ<8(D}~9zWu#5z-hAmeio!_3`KjY=_2Q+NBw6Sw`{wRJ)Ya0A$Al?#%-~%Q z=)w0sL4t-x%GnfCcxBa$2jeS1?6!BC=zbx|!KL16_(WMdmg#jPb8e8TJIn0cqP0j@ zpW}>9NhN2s$E;-6i^dy;O(|zFJ^@=b{1yulXLDN(r_c?&)i(UFxeG*HcjgQJ%B(&e zLrvZq{$^KOD7qRQo$6SXR8f#rVXs%y&BZpnz4gb0b;N#sqS7~;%aaBMdA6fh?b^*t zM&DeY{lTW!+e(IgGb{60SOoKl@@SfT#&#x)i(B%H=|6%ZC~zPyEEWl!!M8jW46x4z z9=g~zYQ(8CnRR8Uo@eIH$>ek|dQlKFUiilsDY_*pJ%y%MG!TldE9zMFWi#XSWTi5W zvUV+7`+;V`4<~wz*XJ?w=vCJ}kASj`Wy9V9Y6@B1s^zr^j_Crw+#n&_Z|%L*m1bLJ z4LbtkT(=P&yQ1TK6yUOXBv60HR>@cuY1raI#OFB~T}bZfn|uX&1b5jHK7J zbJ@nPL~2&r22+0KF?r-CxtXoRgcA3^20_u}+2%;9rC+6=+iqM!Pw?{Y z_Wz(;jphzu4k0Huahr8YJpZ@?Ji33i80v}{-W4|Q&sc4o=?1%oQeTS)UOFzFtzQ(P zlF-Wy!y*pN#y$vsi|)pT?pu))e1!wP&2+l)WE`3cF-G7&6q#+o7CK`mJ{V|~Rb&!W zN8EXK&b$8gDG7=8nsR@PKsm9yr$?ssXt(CK%o?5 z?by05kQm732Oi*$WK5F1LjqL=l9;i=G>~#*ZKuz{vs#>HBwQQ{I>BGY8cNO^y-P&0 z79>nCM?Vkn&L42sq8xBeR0>{OyIFmtm76o8e{;>zpBv@c2a|4Oi=`3&=C)#Vj5IYG zFVufPXj3XRKk+AWQ*1?TMP;-xfVPkQHuqFlTX}>=qr66KGOt};0{aakc;X1AKUAoL zZ}NOgKeq(L*@ETd$ovns;+Ke{uTdcf`vlPu)kJ8q!TGG^>z#4YR>xY1(S0k+TajIS z`FoWH4nb#v0R-Q&Zel2T?$UHYRcn0n3L+*9-C7>eUw67^=W?}VM-C&?48n4Y*7Q)NaMVQekI4` z6^>a`KvDSX^_}KedO_C7s>DU{^`9_ch=^>A7g+f^I-~bmluADG8rrzK!Serx;1ko( zeY7^;b?v&iu7{R)v!1PED}bRu)s};azqt4KVTkLKv~=j9tjSOQDY%SBcMrZ6&Z?oB zW+BRbFxYL9YniF8^0-_6Z7RnxIS0#QCf^C0f@Bu85ywh@wnUUhEM=&=rhQH0tbS^- zcO?X{XN-6WS92}ii?k# z$bt%scw0aBFLg~Elyo93V#n^XF~&QnpQGbdT(FW5kaR^F>UkQCwJ{M^jT6~JlsL2V z6lpD573fw1gR)R7hPw%L<1x=ts-6`*ZELJEqDuR;xisuW*gi&8b)9W>KPjc|(+C{3 zHjQNlc!TDBN-BF)2^+efDz$9ioSDcbkhiF6(tL^H+O&$<_!s1}9S-=Mwje&R+ES-N z?UAPTYX8BS;yc+NW~4s{?GY5Lgi?#RMPb_j%=)0ABAN_a=J{msRYs;p_Ci3=zwdk8 zS6YpEG+08NH1a=0rXkw zA?LaC`>y4aP270Eh9ZYqx@&DB%HZQ&q>C!`Z`y7x&v7Bi#GE_00Eh^2x2O|p=L^(m zwDK-vw|YK(`ZPn#AumdxF#QEFQYj$DCW*RDRp?_a3JGMsrZ>KzPq#$C#zJIIhPggB( zw6A@?GsW?k`Ty7UX*;Sk1w=|0e}4P+?25T}mGr42xK3zHLq7YJV;R-A#cBd`GWa$^u({kC)C*ty>4o_~KIW zes(%5;$C_IEn_NsyZ8HY)BKztN$G#Y9s^A`S|~R|?{zOzu||Dm!O1f5X zlsx5^Kd-ChQ-29)2C#m!H#3p3Jp(FV_otU6oH)t^?4beA+4BFk|MUI8`Md7bWG`91 z9@y3jE-PHH^{vz6(;uVTrrOusac7f6vp5 z+@E&nb04p2Ro&k*Bfs8|x7EuY$cOI;0#%?Q>*6XOUj`ZTu)99?MA}Yb~_-^UvXTFD-xX(Le-*)$Z+rKOHw#f0`Tk$GFkmG;Ntgfk1SW<9PNp|?4+A;k zA>m@{;$}vN%-EAEv0uG0noS{lQ((l5cPnyCenl^Ty!86x)l1TqfX$(!oBMxj15=OK z9?r)XHEXy1y0$M}EPtD=?F7|oL&nd*7E?iK@Z+xZ@@HbADHXdW<*)tuYds$aaCk|9 z1z2du|G(M@OqR3X={6^b%-{sZ#OHKi;|ACkfA{n2?Qs7+#zj9*r91~VXdVMAw8?Ar zfWu=tA>Zw8ZU9y>ca0c6ADdGJbk-Kjua9}bjW=moAD(U7fW5Ilf7e!oy?M&Oz}oET z;usR@&K({;ZB2YKZz-^_(`7c`2i^-3-`Di&RMRW3W?-%1a{?r|26FRO3v`r;ZuE=z=Ctv^o literal 0 HcmV?d00001 diff --git a/src/_assets/img/performance/trace-transactions-spans-concrete-example.png b/src/_assets/img/performance/trace-transactions-spans-concrete-example.png new file mode 100644 index 0000000000000000000000000000000000000000..9676b841008911d5bd4e66d4a853a10d52fa9592 GIT binary patch literal 44707 zcmeFZbySq!+c!Fhh_ng_f`mZ`3P^V-B`qP+AT8Zpq9832BaIB*-6>L%Lw8Cy3`5VH zJ>T!|Iq$R9d*1Wk`RA;MwQywSzVChQy|3$YeXiIOqM{^2ginPJfk24f$V#e0Ah@?7 zkej%-Z-95qmGcw83!c5KmJ-W7|oKOhd@2J zjQbkQRv5OG9`fuFAKsU__Y|KCs|+hkW6#Fwi6S4bipKtk%c6Z@^N?HqrMDwPS!_5qq{sHJb#ZQ`3> zp`(+}^2)~81R)S-We$9f_niGDmBr`DOE^K3Ke*5<;CD6`8zO!R&rn)iAq3X&hK< zNCpNrBQb9dIX-$#kPMmkfk4uk!1Dv8crk$a&kYBX85rUI%IVTWf?z2T#;JNj%}Lm4 z>QmM3sQJ70u2pI}gQ-*A8Bw`HgT*v{UD#?#IwRy+3Hv@LRnvJz9bSdtei=MzEXsIS z-bmCEET&%UNJx%LlYQz~MjlDnKpLJpH^r3{CQLau#h+crlOp>xQD>x$2<*qQIg1G7 z$vY>ji{L1w`jvoZY%jsXaMZ(ZaO^1NTQ~e|bo*5?xsd4Vx_bsh`(#Jw_0zF)6dRg4 zvGEEU?9;t}uM~dIVSoyQ?lyu)L^Y{A@V-1F$oa=Be8pVB>2p|~V#HCuSv35~X?BW# z?T|t#WZpoi>V)+8)OC!1Bdb1i{jo-_dA~LB_wj>g_Xih*^2%S-%LBp6C|LMfG^p4b zw7})xkLRCFS{pX0YaTl4Ng*0Py)I(zcfJtD<{<`hw*yDzADvgg%n>bnbpU@OrYBIg zH~`&Z@vV1Urh74U+|lcL|B6}Lo67cTcKR^wM+={Iw7O6gPojHJ#I%rT2L=0#0OfT$ zSRFqXyMB2Wa-7o^FqAmzjS%AxY);~+7JIqfQEY5CS}@vcNe+DRM#+)Z&|w7?st0Q! znBf|^Nr~9}>K#8wP*F}I-dVoq*IERVYpJhcY-Hbf*0O1_Y4I(K*q!%cU5vlN*vLA) zj)%rMJ=6>arXWP!<^0b?IutnYTf51Mz7SdofX!aI4qM-ETH31l^lNx+9FgXm!+ZOT zwtGT8Xh!UgM{>vUss8uD`)<6E9)_kGngNEBxGfpn$4w>8_e~=Ei#9nLu7FYqDmCp# zr4=VrW@iA6YN^_ZSZu=x`hFzQZ#etFJJ|3cm{e#2SVE%`AXLh$<_d?twVF=pUTT=g zcKd^6cpM+A92XFUsQecYBfCZ}ko$P5iuBj!0eJIzy!--2GUAQwTNArhvS~9pHynd= zFkda33bl6#f*TjKk)3fhefZ5jS6|cw2Q>C8x(xUkW_c8(+UrN@sfosgi7E!#XmNpB zK4zlCF`S%1yxUAeFZLA=WVn8`;{VdP95~Yc8cfKdl45>-j9;+vnDjU)qd_6xT1U7g zZ)N9bgDp&GM-^-+^R9PZ@p2e8_a3}w$9&M5jagE}%j}pWYJT3kGT-@jCosSCZj8%uJ@i&PkP9Jiv>YXmr zyt00{ZrO+%u9H{9yYnt5SiKeaiF^uDL<;2cxwmh<%eyqc9eKB1*g)GS&CI4eqc~0)JbO|Y4fV1p~Hn4D)#rJr)|v`e0de(j7+ny1n^fWVD!*X-=_JNu~)x>geaAdLn=PQ2 zX{)5sSCw9Ep6PlvM_|cafvOfjckp_7{mwQrq1!A2iBpX&P1!EqKeSW9IDLFxLd0#4 zGuzmJ&S7+VcXS-=@IWVA74Tof9{}0zJTYjrUeAWbrLs%0u#txR&RXYjr6Q=_KYC}` zb-n^ny26~-O)%_gi^1!c9&eLI{bs-~sy^!C`Ef;qY|-45Y0vhtnO`iUpVSC=&%lO} zB&)M`Ed>?STm#wLA5U4pEx@Qs+X_I@LKE^35JGvltGI}rJsF~e{Wb%WMzxQX)-}U= z)5sXM?|wnaoc&2Pz=K=;(1FycD**q(l~-{O=yoK?=B~XAe0Bjw?@V@}hL5S}w3`pX zOM}=zYTDb{RUkO(dV=r8p2}D4w~6P#YFkdnCNiR?f*pRSPmLcQVuk`J&ZpZ{L?Q#w z*8T&7Q(dw=gseqO$8&ocs!v3GZ0CnaTTkOExJ@De?KQ2avu1SK7+Pu%=1Oq$i)#9fme__zyTrVYjUn!`0v6MHkCuH=`c0Br_f{oA{rT}3f6<-(hQZf~H0@Y(5wkAbhdbG-j;f|QUG?y` z@#CA-!7D?9dKggq1{Q88i^SEqB((bH0)Q2u+Ba*!jPg=$0(;U2{!Sd4F*gYywC*T= z$_PSb^WOR7`z>P-(Q!29QsVgUd0T1uyyu&qO(gcLrKofev*lvIJUJrtEo4J!=mQG4 zDLTKU*z`>mi^HoXX!3+bn<7B6C0P(dby38Bxuv&el~e6HW`hUbHIe}DHrcWyh-`BGTTI|)?dJwMqB;mX?vujW;ZI5N$f!I_n?irRa~AnMlS1Ki1oJ!Ds#hc zPh&#VS;(7@5lhBD=*i^G~JFZY*-?IjT9jgPot{|lh^EmhGZSrgJc=Sv-H!MGO!RDzUdhAN-GY@G`r zcIzxS3Yymk$+I&O6nJBLY86`3|Cl-D#cBjWnDIZo^~WNO7rQxS!IB~upLr|~=AVId zeiTQzEge;Dyye21XbE-6@^6djz4c`a*n8+FK=$aRD&Qu9aU#hInb%t+gkPS#*?`{| zP8Bmg*i`%$VWts&BanRoI?Ru4zl@P?vdpyNC@g|X-|*{+>8Foue?MXc_E8f-q+SC~ zc<|jw$71zP>z(a|nMe~di2s>v-8fih$$&@S zUnT0_*6&%pC&p|g|MXso>9m;W!gs~J?o40k;oFOsT;^lX-*e|M>FiC;A0Ylv2+jLV ztx|ZKj90!H(}*c3NIb3DnVC?nDMM|>6>Rg4yB%AQulx~pI>+HMS1h2^L`G}z2vzer zx2*i#$|rz6C$tr7I_jXAp%%2QFoY*3RDAq2bGmTq&NoQwQ>ip&Q)p1(`I?XVcR!Cj z`}v;}O@@0RJOzBZr(b54MIS;`3c>b5UdBx=lyCoJauvIhH1o4+yPcdcu-3wO55|e> z7_v^BlP09@*vgrC&hhegW29C877 z!@O(PgkW~YPn@u|tE(PzID+dSLMPq_NG3S%H%kMf)~!H%Zxd|;rti}CFGh=(aoZ)FMYIq~wtx2FJJ^TA-kM5V z57=TXQ*w*p?A27(h}9pnVlt}=R7#)n+cmcn#zJ_fu}NEywpWA?-vwN zwP5>&(0kcR`y{M_KywT(zs&p!KZ5_}67&1JiAE0j;ngGy6fo-YA`-KpK_o{&`&yc= z<_){(J-l63#yRZtN-YLHNjcTaJqYsWOdN}@)f7xn^fk1go4afqR;-qi4}sDF z{|adbW(T2X6I7H^ZdhL4GS$rN72~+;1tbpny`#pi|FH0N{fdkjfgiU(T$l&%G%oxB zLb*g&Po+)^C^zbocAp>5$I(Tj62+|`^%|_HLgixRrR32s1_E~7a(j+(h|T=OQOqSR zK_EPmrvg6kwoSx^dYjJY@1~N@t1mObz0P~qcE(knKp%}PKUS7$RO(l)$mPwUD{F9F zflUE3m3#8F`2~b*#Xn6zO2&4@_)n#%=c^9?y2ZQMX9c2*RAmV`S*fNF?kC= zojtTp%2}0AXP+kQnpkJu99d5HW#~}*Hoh@CVF?x4F0Xtv5Yfh_TYf-Y^PKhl4W9Q%1h~B%kd2O-x~WYl=SvX_e?P#$O9$JNHi1xj8{Qu`ozC;k@hjR&E>k1X;r zJ#BBfJy_{{@xaONh;J?@BRay$Y_6gays;!gAje79?^c|{lu@_j(7^E}V@48s67G4j zk$GuLl^%Q&1;8Z3=@?f9k+&`CE{m}V4sTrm5!?ya03FCjd)|e!ZvP0q1+=yY;_ALX z7Jwr|t{-W5cg$X$;(4NzrujA#1ky3M4ictM3}%XhUKpTl-4i#Dh9A@{yyx&~_S!ru z*E)S}aGJpWMQ=w;XzgHMTgK}UA$ccD>fMrc+L->o*mgOUpUbqG1~wi6>=@@u^2GC| zOhcp_S|gnGaz%EYib%S}*BNhg&F960?^5-mjKl!t!$HmBmaWY=_Mk^A=4>tWgse%Y zRkKygZ7Jt;dbK8REAceYCtIx8p%ia6ta?XCSUN z^XqMB(XO~$V!|IMzdq=nh!D6wBM-wFUNz;iMPC#c)G^dv{xz{Rdb8HiZdsC+TDuvv zFC#>TW2ybEjc4Dg$NXaft})h7YYlXP8tHa%@uFAMZg1JK6LON)A}w1n6rU_N|v2iJb?jverRHbPi?FIx+TcQOV$DHm_?-D08MZ4Zzd z*qh|3_v#1)79+yux7T(TxQjaq@nQfZZO3_WAfcau^%UmuXp?ECOPKOl@q;`91- zPH^G+XJu`Wmd_5^>{4=b!ME}$E9LG_%p&?MRfC}xxu6E`OYzMCG?xY;W_tG{-suv@ z)<>MUmAG@f-*I$w88bhUcSY`&1tL%Nt}#~-jr8@5k+xqFa4rBpv&B1(yHX0UrXecw zb({0a1{-iUlj1nACXv!yyV>#`pyxph#eoiG5dTM*d^M$|cFm}a7Xvx}?jPmTb#Es2 zh7Swy40v7qgWW6{I*&EDOlpFzo(lj(;No;jEtD$9IX8q5=d3ML8kz{yKSqLgVo-<5 z4@gm@lMFq_|6?p*%#FTQ2v*<{KD4ihlC0jRYpzIS4Xr*@?;0hs>F8o zVy6284ANTqk1TI7P9=%ecXsSSRp{38;^cFx4 z+xG+y5Hc6YBQM+)4;blIi$ia@{eV|Vp;ptQotrKL8FfY{8?Y$hxqej$jaQYMT^CXW z+ha{{yC-0Ga^8NI!ZxhB4?kX__PK+^og`^1SuB4ekePsRIF9bhU#=TJI#7luMg(t!FR0 zw7a6T9jTO?CQy*glyWF^vAcC*lz(|FlX`w0!}>|}4{*xU{Xd-5(go?;C(T8KOTlua zwHzf$gzBlRC$%^Y(2t$pfCS26BziUKce>OW$L`|CMt*F5iZ+qYNxORsW*k#@*62H+ z#m%?zigu$`!Q43#IUuWmadiY4mGp@_XWbaCK^ZPRF%>TCvfk?{omR%wrtAF5hkI^2 z31}Vo_`UsK06N7@sCw(pAafK`i^)<>xS8hg+JS+x8r=JP*Ua5tqRO&MioZM=Yh;a}*U?6wNx-+z5}6}Ld>$GI7}S_6k|`9X*vg!e$5rL}b8 zOu1$5n1sDHGVn!x&>A3O7{ANmMsUISr6aZfQagiM#%=*H69A%bDC1Y#`w1|0L|Y`fg|+5vty(7Et~ z?i3I(zu;>T{X(NvN8nPV#Lnbcq+5}D>3W#Ut*((89%&&@M6@}E-W^x~J*~6v`9<$R z_=hF%)dXOlsF~O3)pcMLjU^1z*7@|eeG$<)?fHWk=mA3*^h2Xv)h5o5uPSg~b3483 zS7X2ODYdrfN*qzKBeCv61}*MjTo2%g0Q#DoyxKFj@&_gVxU>hU%bfbWY*YoCC~;Pb zK{MNTJUEb`A0W>2Un)W&gSlSddb;BmB}I*-3UIf1?-sM zq5{{ZKdRxDMX46|g7y5wa1719{^24_gq4Db4k?6PPro*&M;0;fw_t2UOsVdJp_ds; zD;SU`!Y9brV(pB>k-y_SqMe^^!lPT8{y|quV$N2*gm?$gB|6bdz@C}`N?_ChK;iR3 zisG5~1_$}K%O_SMsX<^4Vh>1A*_xkE)45y8>DAecYs*}Hmcf(-aHQ39cEJm_gPnV) zJq?<|?sF8j+uRObV5u>VMW~+{P&z-P+Qb5eFJm{SKN~R@bMfg)yLCMTreN6R&TWDF z4`#j$4CCdTp3q{jjyp_8d30s;gM>G0y6sm7AkdiL4z%{DRM8`G{wPiVTfFtlVA z1u(b{Nl@c*vI87)pCX@kiy;&fCtx__vt69^$pUPG;G!a{rr2k5Z1Dh%ltFu-1S&$&>W;yQj<`voi{k##g?oeO1$`V? zh-Ai*TG|3_345_)<~W@XMzG1Ekxq;$0BUvLto8vyZHVst2 z)GI}D1(PGs!*cB4CLp9~e!)GpP7@|!XZ%|<+-8AIri5655+Cv;1f(w@?PICv^Fbe> zBfuF8?1XHa50MARfMnpH{f$7n7{Z~;Q{z$ z!4u*!-u(SXA0P9!bmii~G zYCh1z9bmL0~4Q>@gn=U@=GqSe!dPSp&NK8 z>U>s^cBT3_dNNg!rn|$(1Z1qHFc&^FZWiYF>b-nJ{N-U7JIEBgUK;w@-ub|P`6mcH z{f%w)QD3evbx83W{IQ}9()r^Q7}fHe1$~a5tq`(8IxtBHX2)QpX`=og8e85*gj{*1 zu!d{)OXCfO7@29-{zQ@8rc>m)<8GP70eQOJC%*VaE5Q`DXK$c!Q9SVwG7^vxf=t#R z<9w|6xJ!Jpu95Y~5^|>;6Qs>^5#xn7IaTlatU0?8Di^1y{>dm*`lZb&`nZdTDMa$| z@x08vkk9~{5EFg-C(zz6me0h~7BnK8YY=WX7v_(UV(3MI2`oy_IP+9b)P$(fqRi7a zX7L46WX+D}M#vnASTtUCiZpl0LcFYcW%E?3R%WM(l78joU#W%>Cev`Ga#`7jtI?J^RI?yQ9iG z{2p*yY$d)=e#b6-nUMolysm_ei&gQ;Y~Smg1ry}5N1n0s8m8Y&tIbrE8HOL&`J%~~V&(-hUvGcedZ%9lv7%8Bsu-Ozrb63^bzB-9Zii;f1()GGXQ1D%JD*?sF zHAj?bx8ZG7xU=9KYR2fz8+&ZRnt z4NLIW*)iO56OmZ{!2IdGkobPP%H59^p1z6_zS9?G9XjX*nQT)JUw;c>fBIF z`YqO<^obicNc2StVd)-N!ogw4)ED1c-CGA{wvfFnMoytO)1sm8l`qV6+$o|seSMKA4oy{XZ zNYfa!2AbT074UfRg~ex8=Nm&$6bDNzXRcjqSA*@EZTmJgnrP~sJ*L8ET4;T}Hkw}U zU(075u4C7qMX)x`h!J3YW%9$i!LbMwN8|{}o=U_=4WhQs#}o*gSr8D2*v{`9%&gVp zb*9lCk-UN|_XK}x#fz_q?fS|`}=Kw-?C2T*27($OSxLb`o_w^)rvZxSdP}Ym@L=Pmc^wa?UC{ zrN?T-r<}$`K{euOK8n;Le?r{~XGA#%2t1WI@a?_fhLs;sJE*Aln+UF7hTO-#)1V6Q z)zhiNr-2S7H5mNVr*t)TLf_r*>Th=9mnmmx-TQ~&iTw>nh56GVa3`YX6yXxjfh;FJ z;p%4JvGHiZdok-86vJSR;MIMF^-{5UXt>pYSPM*QSTH<>Y3G-4oOorWc1lXGt+ zN8v2cYH&Z>#8G{7!$UKb{Bz)IFAFKm8fa9Jrk!u+n85r2Ie_?Q3|X=Wg6IATqe#0t z!~yQ^Q=qk{Z~tC!>SPUD`NONW_&6g&U7B>3{(TU>cX>&E+TKLV(uN z%qIMKn0U|`7Ewj<+eXQ!ziy;cS>Qgd*zsHr(pG4^wgoJ*SS|9(KJYiVb7P{Hu{FL|aw9J~`eiZ=D>@ZH6iml4Fcpv?hM`n5!ngZVeEpiSAs364# zzmvVCwE2k~Gg?J^gm3BS@o$0`4C~)+tmoY6N@rjWi=XA=P+P&1)y0%Qo z^>m!tjkjTW7DH|u9hbhx!G0%s_A76*V1adif$~ix3%#gq`Dv@%7CDFJp5EBpAgyxq z*y7h{$#?xw+YdyIV4=|yDG|)I6B+d^1ykuc&JQ$Y3$znTT|9e6ns25=NcZ#fN$g~q ztyg=t$dY(O8s|@X%H5?p$(lhrWHd`?kRnlA=hD})l4-%%z=zBQ&kN2cD>ph7XA*xs z!RtHw+jm6;>)H~_|5)&A#%tjt)nT2$O#l^Ln!GFF_DLXEsIkr4@7dqbvz!^b^Nt%h zn3p-{Kd2y-J)CweGgsX)HZ9i^*4qKJsb=+!&f_AzLIWxLo|l;1T$CAX)r+@9thv4@ zW6QlE@*&(29#pTSS`^n6k4AF_2rkghm3cGeW%=hSm2L!{RaDlkdhc~Q#`Ru`G!tPn`JdViGg zl>OWbT;Hnt^*oEm(Hj~pRJY6F&K{QT${hnlWu97JT*5+#>vWhCd8KRilfKAPdaorWL=%Gxi5 z@Tp$on8vl$7)e<5cV&~0ehzwOqDi{z_2H;-YvU8P(XYuFrw1B>^~j9r0|(c_8QO=c zTBcLsVP;4%hsacy65{8=`9LL5=f&%Rb;kg%-h&kigyjpkO-H)SfUkyKgw~hn`>vKoaOLxzq`aFaV%YcZyb|3$i_qs0pr7Gv;hLwOiP8 zby{X}>XyPaG?{cmqjz4jPx=#-LSA=RPM z$=Ec%&MyMe++MCubjq=DxAt+i?6rqd?4dNedX4~M-|{d3^8i{Z@maI<<~q`4Mo82! z2j47KbFcu_`Vl*N>Y_;mYVR_-=%C-Y&nM$LIf}BW{bq$eLAFNHXw0SB$0BX5Jl9In z67LkgIv7A5+>PwJ7P}wLi|`1zLD`IP+TYpO3-nz(KsNEA_3Xc~-7&zOi2}IS^|%l$ z;vG}+OnkxFkP!v_{P&(kL`;v#rC(1BK2s!CtNrNF-wKS9lHiI0+%#6&pZJxYqj$b} z;8q=j+`AJ_9EAMP4s(eX8nKB`c+$$U(M3T_c|#pf4h>#`#K*vCJ^Xc;yE8{ z-?fo>(yx5N-MDQgcR6pP$Ug7(_hc&ADFpePh1tNy(mKzes%k_F*)*0WQ7qu5-7EBB zsSIai>(Wn_;|Mf^d2Us*iq0VlcrM3jESVS1{yHUG7tuAJPp3Zj2erYYPQNC0!4Tik zzlX-C>(vXk4&6JUjB2}^phU~^>`fG{j>GID%cKhf$0_84)4F7j)3SJ)<88_1l-j5p zDzSh_AjOVM79y=N`DqV1fXT#mQ6E7gkm7~qt4xIB)hHfLRRVmLV|tN!)RKj25;2-h zKSWsHgsU^|UD%MV(4^7w(W)}Cie&vBulK&Qz(l;;tu?nR(T9@&iE)x}SVITuoNWP_ z0H9jm_gWLf#W{A@tR@VODV9E8{I_s;`r7W`KTZmkvY}=ylX8_6#Xux-uFidzEIg;9YCDrU~*yU^b1{Vdk)r^+0vRs2KWpZh%f~ z$`Wg`bebGZirj*}(DWN_s`;^IsXcqP5mZTkx@|0FZ~d9L!$+@8gG+B+Q?7vxv_MvG z)C(fIL6p2vk!b))tY9+_$7_Fd+A(GESA&}(_tU^(B*8&9JC0N{qxu4 zpZz6xsh(?2Yl!lfu5a_UGA-6onG`N_gC)A0iSEFc!SQ06)K_oK&EJdGd zJ9RY)eT1~YvDl91n+L(ZAy2{Z=0EX%XLcg?GusCtVe3jyuF11vCul`;` zsI0jxW_%YuHHGUn{>wt$l1VBX?gNM-Z>iP{0IWa`XTY-_q$NK{RUg!(*v$o=8%!~9 zOnzNg(5jp)*hyg2v+!OW@bqZ&Jdz!H-0N%7^W8w{E2CQBl^CvVza|%pd|jjW znCL+pgm-mf=Si5<52(!2obIyRvq2}j>bs1kJ(r`gxsUggeorAM(7uvmSMJtJ3sK?z zET=qmmK3D~$*A{a%-TMwGIWI!WO`34?K0>L=~z;R24$$(kj-OcW?#A35~s?%E-}uf zA)fA|2qjzO1ER%zKugutTm=J#>PyYW`a(jmh@=C*NCs{m!=D|@2yr`X)d5G;J|=%m zatSYu^QIffveX#?P}sH$h&S0y*qDqTnM=kkkFNS%p`((|_P#z-&sI}{eg&5G-5?VK zF`Yr?ziE{CB-BOILh;~^7>cPlZy2tj*^h3DQs$c`lcm-9z@4DLpqht#r$9p>9Y~im zvRmV{ss1>H^V`!VeE;ZN8_m~`;U$LL=-UqJ=5-g-++JR6L@r+UWN<5hcMGZ) zG5RV1xNipONQPM}~YJ1C8z3w?q=ZQe#pYH1jjFG3wj~ zd=BjQ8rUB+nO7f8m=5xP4{QJML)0Uqi?&S(z-g;F0rrp&+ht!8b!# zK`kKllI(f7j7pYvR#MiH^3xF@dIHP~mvcyN@BRieskXrWvbVn9R&98^OCV&Vz-?^jSq0}GE$7%W9a&r zI4n=r#F~L~u{F`-IT0TrXwz=deAIJYj(leXj-Lk{xCC4slK2!={c)Ek0d2H@4-P)P zZv?sz@bcVoG@s+87+|rRfdQ63nj%?fXqpW{c;EqofJ?DOl`DXzuQPozoaH#=pr5`K zRH$<&5Uj!1E8xhaP3MfK6QgT5Ua z@@zvr%gM@|^ow#ZESQ(kE6sOW_t$KV?BCJ%83j(-b~2gY7vBHJdDYb0F*yQtxXw4m zDFqIY?wje$KqovUF70aI)U-~AdMU1Q!hUV=QrQB<8MPc2m<*gPwHyRog!ub{p~rY+ z$>%DPCSChY*c9vUZ=gNs>tBkmWXAU1Da9{6pHnL`a=mhmJm7>#r-O8<9*M}?rk&8{ zlxdJ;kfiXo6UkzaJG#vC4LvU-R}kpsEp_ zPHpwcNq$qN_V0j)8qra#N=w!{&;|PpyfiDboQ*QCRn+`meVgaTVt_GGBtC6EwUJI{ z@!Fi-S=Wh?o{3jjJ}hb$P~ts54A*ell!48xJM}HZX~erxg%h{YPn1kk_2uomp8ybR z-h{f)tWVT%S6}VhU6%j>itu0EU*T{sqyCG+qDn zr@RFtMt=043MyV6NzmL|ggYUEI11_-$4%KLk*? zNaBdtlA#uK5%X_*>r#OgZU_QuO4?t2>fOYxviHb>Q-Kxvo$sXK|Iyxf6>5%&<78al+E)EcB}FiRxli>O%S(5`7L13fqh7HS{B;;1n!y}Bra z?9vSD_y>HjWC1x|rAE*M6RRZlL^zzD+GbAGZx-8X zX6#@KY8R57F@?x=8x$Vop~(iw_p5wdQQ2neW0i_O3Uug#GX`X!fy2AcGABp|lR!9` zm_4{Ca&sk*s^0xL>AT;KN$mS<>*MYBWy|o4>wi<+E*H^N6xC)+w0ltbVy_*GkWp{7>EVnOi;a^nY zOT}4bru=I^e~O;IyAhO%sf9quqCv__W=~81xW6Q`SKiBA9ZY@gjnio8jUiSR%xN@c zK}-?hiZ1zh&xxJ)DvF7u5oM+s41J(CuB}z!w@erYF4g%rtMazKaVOUA^w5DK#mlfO zwZ`Cm%QXeN6v@%SMtOB;e%}wy31a`|e-5{)j7lpOKD>qtT#h&Gb7FD$|G)HZ0y+QN zL6?6{Lok1=Q{eUDU!lU{CX0{vbqD-!1TrNkR8QQ?{+{Q2ky&Q^hqadoDGRc75W~{XOC4x; z`z(n_t~TcDyHCkHcKMv@bU71ZBW?sfbsy~U6FFKV^`Q8WiJ1%s^QXyIt`Y|y4oc)M zo0*-?!avz)=hH=qktU634(3!qoKomfKW#(W~%0a+_fN79;Yog!i6J8%LwnVNt+9_tAJG@$;g;fHcD@#TxXXJIkL4HCp)s1P}FjT2rr$}UvW+KnB z1D)4G>`x89;_Qp5!tY}0Ypsjl!*2N8H-lke;}O2h+vvvhcNvyeLIFREQUM9;PjS+E z0nsevr$GC_&+`8lrzczQSo)emf8GLsg6Xl{Xc&HAj__VM)%WgI!N#=WW7wh|NQ|#v z(z+j~oQ^kIRt8WCdh@T8cg_9`3|^`!0>D*5ITY@v1aNtPns)qp-c|cvq;6dK^9+NYkIJvlRmS@l2G)#koSD0Ko zMo3WU!xY-ipMW;p^TH8jePJI3jk>lR@t#_TRi_*(a_JvHgBW{P9hWpU-AxE8;NJbr zV3L^odqJJS7nJq9E^lA*?M=75jq$8MQt{f+=dDJx$FQJwD~CyxQ|DR}B+^-<*HdRWSMZ_WTGT4S(3) z5xqR5&N+t4(PNA5HC4=r3Ny(Uxu=+=w#)au{4FG%5_rT520i_E=XuK9qUXs*CI=?k z#N5T+c@C@caqO^F3u8C3qEqKrVaBO5m4Nn+?*e+f|YN)LNlg8~&K5*=7bTv3?I-lXR2R&rtotEF zuSqy%aVM5d7P}jo8wAB2;Ije=9^7`D+yHb*4&0#uQGb}YDA0=y*P83#J3}O z&?$twwZOCEytO#f??XHUWR*WbS264HP7mE^R|geFVspn&@W*!wE$u%LNJlZQg?@bi z=}^NWBEtjPG4#OalmY%ahPsc>%X!hG_1w{ z1lQ$F>bWhg+(?=#6_jVei!Nsll`uN{=qJEn3KNfAlZU7{GbRgpt(`WjAE${<%wW%; z|FCJ2QZ`3s8iJPn0riW=4mT>`vMh<{C#*Ssjxze{Z#uffF~=yR@{dRcH*I)XC~<-s zkdv>DvzFi9B!^AyL`#{I-yKojWzW&&Mb9tq;_$x$mvL-M~XGUagV&e|cm zrDa0tySKsOrsu}_x(@?f@bvu*kdKWS@ZNc<)Kw`P9KXP|Zj&eTV=eTe+8UI5H{|NHs+*xJ94mr%} z;rS@hu7-3FSFgSE{l7kB=Zvgn2#1Kebwj}a)a+)b{(7)9XG)IU(q>-26 zs?+6#A|5kSN#NNHGhoR+6}?*8YEK9cp#1n>{vq;($TR(QV(~t(oQ3`<#E>+BW45E zY1Hx?S}mc|?rFkehTr-a^MR0sS@2SgKF`iHWcn2?%#H4o`WDJq9&+Vbu}l7&LE^-o z4Jr~K(_tU2kR5powtMK=Y+uQL|0^Uk2&3A}_oJ=3(@ei|DCLr-hTiH8$yVJh2+N*k zl^&@>YPTR)rNa!}nO)nc9!ex^z^OJ&9vekyFH&N@VM0YRgH@05FU8?~;H@+%`*Z2D zm46LtXJ&|F{qqGBQY?MafuUm15Y)HO8#w>c=#sy0S-x(P(P<>*0?18L`hrr97-LE$Y9nvp2G~gWcR4zW+gikNwNdS1LP= z{^F|ZpaAGNnXIM(a(Egp1LwX$WifKAqoSAUyE_+Sz7Tu9>wMbAWu#p9Q_q9)nF5T^ z49<7MwnlmD(~qA!ut|qMPWfoHWo5M$apNz=-&Kmg4pRQ=!#XV*C8#)-!H1sPe6}S( z%lzjlBJ0)lCIUUbO%l8ef#_*!#CbZ>etMhPHa|y>OY(54p&=u$tOKzjfVEJI*D;c< zOvU%}hdg$sw8}}9$E{;655X@&TKQBL1{(d2b|OHKOv&2ynS{_=E?*)!GDc1!)nlzk zqi;(`tBz?f`88dgY~{zmX)E)}O!xOfn;KX<6`50O?4^z1_H1{C ztv{3yz_a}GnO*$sRW5DR+DX2CiOjZFjQjvNd`<83%Eur&w7RBd+w+@z0OcnMy%93i zw=$0@rM7k^tM?~*4Wf>hh-q_$&)ONp43ImOo1}Hp;&$Iwlr@Mte&*Lkx12tEx|PTR zP8-V>*$--JerhxH^7$H1hQ_QLg8sv&2>WY0Zrl;Vbmh6{-6sWTZrTP!~sEyw-PZ0Vck@ci^!yikHH#d%5 zh<}l4?^0wQz&mS?*+@FwEBL%HH(6`&3s}?*n-BE#?NV@_RE7a5j!^Z+znDnB~z~vj2XUnCIJOhy6WpMvlJjH^PR265-CxkleM|Np?5$`=s9+A|kTg zZfT8M7?{fZswuM#?N#B!~A0!=W zfM(wHt9Ntjn1utQ^1!jmoFE=+P%d5`z-R;!@ zoB8!@gyv5(&(CAezj@}gk*+uCsh&`uSbS93o}@~dmJ`xYdjpyXf!^|97X9@L>=!k5 zGt|be-o6t0Zf-i$Wi|#odix8LXY*~?aY+llkEWHM*pis~!~&VD^u0t=9TL6VG=5m?@DWD5v~Ih$lTq$ZLq{xho5cx@JoF58qIDx`QYbCwnWg#y;q!X>Wka z*usXR=O`e+>g{3YJ1%FN(#e&bG?B*IS#f4Y<*t@+BYY6I}56{oxDzt5YfB)LG z6J+WWh!BJYwVci1SQ*T-*qz>NZEHmN9;VG2kod+4?cGl$j&xh~cDn4a{k+&bW9#A- zhgr9YCzMj5Un7D(^s$u`3tImE@_{n%r5Tr!DE{#6&FMYDh1F0HrGdF0 z+VCgy?BmCg9z>+5=r-7-=}x=j$Gwj(>>9*7E5ZerbY}LDQe`>h9g^Ys)11)pAZDUO zqbe4Oa9cjw)h!>Mtf~sl@S)G4wkc^{gQ}xIhE|vNKwG(NE`IwhoI47ir%=AW2+SjJ zlhrb@2D>IB?C2|z`pnSgqjj`#SF*oyco!KuV>d3%fI}9=n>zM3M=_1^(Bd4&BO9I# zv@IJuk@0eW&WC>46ZYYUO{Ej=BBfTK?N_w=G`oJ@lrw##(63Iml&kE$1<{C>t0~w6 zHq7^<`~YB>bwqHa@3VW}Z`X1FY|pJO*TP2e8f+YDey;J(;@S6Slr&yGLt1(+N1JqP9>5eGBGqvgIi_Go4KA-JMESMR%^p(5 z{GC}b3lFbX2W(S_{viy89qOK$5=QWYUk5jlz@l(T)TL*?ciSI`uO5r(U2j;uzxCT|ud~}P=|PRhL24UeGG@UPaD1g{ zSA8ET0fO*Pu?a#$Di6jE#MAlFwfnIjPRjtUWVv zE!Tf>I{wM1cGOYlO9_{b_Ti4o$mL}Ew3?=(-1mC^iy9yc2q4NK(iG3|ChvdcG!iUy zbo^@XwhD|N80yA$X!7*H(AZu}i({^9L%Ia$xv@kc2YaaxX?7D2Bv>lJc-n~EbAL*f zWqD)zI$b?|*lJa=F;U^tvZ7y`FY<*L>7Rji%V@#;4U6Zrx3>uwj@ChR2tp6qqipp0)%I2<~E8e3xGD|_!Brq;V9SH8MgVBt?p^QH|pCe33|E8qorJCuR*WZi{lb#>hhfL1Zh)ky$rky%oIb_qSqJy@-I1tF5}<=yV*!9_q##^8 z;h0oU@CD4v*v!XLZ2$<)vK!VbHpNg^xh^8#8S~Tnu^({x9_vxwz;kGv? z6Fqcd%O9Pbo$cNFF>f3_Ne8K6)M+QomGFrs;gdBXqL?t4Dbt&rF&3PtI`>=)pX5c? ziJtEP04*jd8bhS6OMejcWX|aTXOQxE<`XJ}rRkL|`I7uo59E-|BLK+=C3@+tYn8hZ zk5=uV;fa&HYOnVy$`IJ+1$oWMyd}QKl*Hwqj~y%U$_cxLv3rh93`YML@4i8Av;`dK zR9gA3U&iGj9?^T_#*dE_s@2`?hsB)(wg6;NVEqb=oI|lHjje)Woi(_xTwMlh7}>Qi z^paKB2^@mJBpU@5CN`!bn-nG*yXL5PZCO=xmf#;-+fN^Vh8LM4ipf zo-b?w?(xn06M$SqOUrS*%kqcQtIS!4HNXoh6ag@Bc!4b3!RegO`cY!o0{$wQ@K9`tYW%snE2 zq(#EJ!7;0Isn;R?5kjKBKK}%b;7} zI;l5E6|GGCFg2RPJv74`@E%}3YAVD`N5&|B%qZcA&w`9D`vWSoR&hApKH6QqWZmR| zekR?^rywGt&T{6fY)y9Ni_R+;dzyDI_%S9w^Y4D@f($(XzIZ@P`0%PffE~-|;B+}- z7&R3sC`i}i_nIAp=R#K;K!s<16Q-FJ1LzA3+==oi0JhqpybGA7NA0HSpbH9;Pod>4 z-*y9(^R4(ABqgDB)be02Q!HZZIkBh+r3W({X`7r7hzS0>N;Xjp)7QcC2q%86d=;~b zl#SO}-Ajr}8xNquVQak5pwK`bT;Ro1ei@%ckH!N9sv7BHj=uO*4c9m_iydr)*KUpj z5x%Tc#^>A z8a|igneF7}F2p3mnAj*LSu*xc4fJZuqNIVK)8X3Rj)AniTTM#caC*)~ z*75J2cs*IXgHjtG;D#%fm&c>3uAr^1z!>CQRgCG*Hb-6h{vw=O1_&t?6j6AA*X&1IpX(G{x~^fSc0-jQQ1t+2#3akroHlB=!aA*4rXRX z5ppl<&+RK_;R`)44er^CQ9s5QK|zT(>t=ub;R)}81tOSm{-_c-49Fu<5Zv&zJs@4} zb!HWM}KeXt<WXu3U5lk5ANiHV|oohVypn5CPPjX}O()GuHmDp~9I0ikG zmgvFyYuyH`uNkIj-+_mHN@=VC7I6@+Z{PjU4#C@39wEG+KC}da*6sdwj#X1abaL|Z zRvtF=vwjjNItn(6K`y^e;(80JMOn;xU`fhn0$Z}+XaNjv2psbM~d<8h; z+<_I(`dPLlIB*A|;e8XXv=E{P!4=PqzXjR}=#WD~?md7a;3~(upGRAknMfQjBR4Pg5O0{Q(p_`Oe3P3MFU#MeAz8GQ7W&BzyPk4Vwxd3BAwhm-2bD3P zE@X%r2ny_5$(*R;V?|e6JmSv;dhU6Xq5$bJ&`(aZ8ye;bpz9^9T(%gT+cZ-gnjsGDx95>MW`#nDhgopl#a?ERT>bGjWK7El7~!J;LzZE@nw%aZJv7|rdl_$ykh{iX z@v(krpIw*;iVsC-G*uLbMaMH{`W?&8-_AampLpGB72ne0&v?0S$Y54!OlH2A@#*@Q z>N=QLrp`4_LiF=%W&G~9E>2weSy@T>OH=#LlVt>WQ~*By?S(n^x{GXqB~upNJK0h8 zX8yry9{GNq^8NT*FAD3dUJbZ=PusKIt@Jn>Qe`kCI$UccZgC>aG&g$*@l9{&S~Rf<*l2(#``%Sw8c@k|&PLnHsm1M2Unt2~ca zb!;Cz@1R^3Y5uZ zhgKg08dmIjZiaie06k~g@e}5cY4_8gWQwa*R5+GxAGB?N=-~s+&62#7JGPGOO`=_A zca`u;end@$sCE+a$7Sm$-m_yuY9ot(_L=6HJ>3Tbp^uySwp;;|8@G-OQj)gJw*m;H zcxyg)W+zT&Fr`F<^zS1c^}Rsq*5}=kZm&<(The(!{qH<;6PigIMF}^)-!e|9dW|`Z zFQcaaP^(5L0tq)lJcN_vyCcN)B_9{D#w2@Rw?S{YK1)XbyVE})PW9-HRn0DL-|kz{ znbCAo%ClRLAmqRRVm~gxVFCnO=?3~BEEplcvfkeMj|Kg=!1h0(DVOxG?jEBi4@#tI zeY=mWD*c|8dEpM>+ExV1&ztR}KW=|p044snV7jmriVDiLNy5RaT+}MxFup$;Me8vJ ze3lZr#Cv9U3u1AT!oc#zHDRvdO2Lj_BRuH(HHo6qg_wfUaI$luIP;&|SxJ6HCzFU~ zsO!%`mID^0t&Ahtuq07(3UU&IRl$1Ja`&W*lAv(x2 z(5$}g@>V(=wqQhDF>3BOVZ^mtCN~qyvWxVyl?1D-#qhIjrM_b>S*WC`q*B>oeQwwP zdsMP_RU(3h=Qkz0hIe$Dvm!g}*CgE?tVbSR*-&ANCWlpbMSF^az9L11?|(2U(N56L z^EGM*5cq+Tf}}iZ0euN?hm%d*3(uCP79BNTQNO!#=PWBJHE@dHN+!6p?M`-nqqkDB zd28GvA6E=%;9P+drvhq9N}8jr7d3H1u$}4`HFw@|C&!RgYGn}-(a|!HTXrrCjl;`c z%Y8igg6{ZLkZs`2M>rGPrZiQS#1B~|Z^M57I_dc$Ub`@tT>TZR=e8E0ax+9Tjw~)yZkQGRCdsOUs*)*)Ak9^fm9{N zclkvBREOF_thic=LOGXh1XDam63TGT+9yISv!`A`y~S)wgFN%xQhJSJijNi6l!8s0 zlJj^jzhuUl|02>#R8kPtZb~y+csQ4ha}_J9q)BiP)=%}OV{k2~61l}v&;2{hS9oW^ zPN}-2v^C0?uX90;2Yt0XV{#(IGb54nm0#Lg76I+M{F=$%JB>=*I_R5pQ!kAx)4bD} zT*^aD5?$U@kWoSxr~(Txzn;!rn@Vc;KoTY95ymar;+%Y`?W?YBZG*!-9bY~p0fgiY z{a~i|W9YQkPO@-F9#u6o<(x-5E6s^7I*F#Tg=+uaBRx@JnP-8xs7Eh9`O79PD!r+? zcH*Z{yWb00s+gX%JDi}hBfUr7Z+y|T;+9xv>^Fbt@@eMhJCU5QVZ+b)dtwt>!97Ipr=K5>HujQzHNyq(M)*}z5-SPwCc}!+GpQJvI?T?vzA8|^ zAQ<9{-(?+td>{H_H-U{8cQ#Q3ru#)h_aKEB8W7rlTyO8}a$<=?-pLJ;fFboqJe7=b z8xSv?9pcSk6-BOE$q)0}=uu5Ue*)>?ZG9!+-z!FT{J; zur!jbhykf@_zy|^pRn}i8~-p<Poxy9~OV*!H zfp5BDeURr#y{GvlbSekQRC&c(a=5A7oMr5jqoFYyksZ}jF3wFE&j@87ri_CMu9o1J z<0C51jI92&;BXoUy@u#N`Rm$Ny>cd4$A2hY1}F68Y51O~i%o!hM>Wxy*6k7=-Nj$m zdybAqLa)DZROS(1mvExTqIhckC8-eW_ALI#C*>5`X&gs!JG-~Rf7h_tBe>-unME0* z7R!bD;h>UIw`bu~OrAa2^yhxkK90AZBaI1Mc3Zsgli4d7LI1iWw2ZT&%RDlZ(HXNX z+xGa6l1yXFYTwL;z6L0=Wo*nx{ekdB-TJ%a0gEkD7;FZltM78K_MzTN8 zuc=s?*RD0=b}6hmt>ib|-o1@~T(_4WaXy4*PT0CynUvKeo<|9maZCh5pB?gV@k`1n zBEF+RZW2I1Y{m~9bymx}oQ6(Lu2X$J-3d{;7O$hr*Vu;D znJMDmB23JnvrkLf+G^0L+WIWVjKV{L@YWh3lqG=>{#zx{FW3B7KbMQ~GI}>d5OEQy zy2d;Xu9~~D4oql|=(r2>Es^xO7IYPwqLFul{Kze^>5@CU7DIV!uO7 zW`06Ao;OYjtbbG!&S1R#vsTyTInSK=Gm{zwCHYjq`lsgaatYE^XbKBdM~1$!H>j+7 z2}}tfo|69ng`pZO+gbwK+n?4ibnGm};t#^8ss>QOwSPZ%@6nEVw^Ow>5ET{Is=aVt&F7l!)*9(G~ zAEH_lG{Fgy@=q-9K%Rd1r=)|)qL0pO?R?A8vcJ)+hu|Eo>BZu{zgS(07bXQb{6WwK zB2;;^E(Vhu0S#iZ$Z=h{Y#O;|ZfU_?Ft3y~YBe~t19v=k0XvvNFL!P5kln1$Lbxut zYb*N4OvW%?86Y}t8^YnIjKoWqtFX`OFEmRe434|kS#1tk#SVgUt@^!%kJ-?b~VE;EEF^h;Fk z<>xwplqZN4I3fetd7mtgGrg^kBn#>I!5+?wnF(at2-9Mx;zKj{?u{K1=XG)-!ON*~ zL}*^MO~uZOI8(37F~c=<$f^LyP6OW;Qgl%s8O0Xl(mB3fx4nWb2x#wI`~khy$0Uoi z?TrMgi4=FZ4{@KrVll_t+S|nU858>~g{~8iF)?T>AKgE;=UAkP7!D70IKpyyXKAIvgEKfXYR4zM6>*PJ{a+_% zWv#lb($%E%TBm-QVIEMD>e(xY#QNO?h9H$cjSS)vuHHK+9Xbk}$X40Bh+nTvp(8JL z7ARjgK;t))P-^_lsH8Of<~sXpk(~Nsz}y=@-E0a{y!8JH7I~Iy(9>44**MkdQK7)k zB;!>rZCNlcTH#@%D<7PCW+pT z%J+(6Hs`a(n3yxRE9bF-@b%6jdjN*{+IO?WCQC4Q<}hM~h83mc>@^4ZqFWaVRA_0x ze44h($u==vd9ME*FXy z5KrNMY^TgV;IR^W@Nz#}Fsr=W>l!E}I6>)O9%83X&HkV}9((8AAUAjw36a6~j?L{; z&wKSp5YM)M`a{g)kh0bVE$Q=-TcKw%T_s`9Vu~Sy{89X?hJE~_M#+hUD_Gg}JrFKTtY2r2+hKwOJU*J%j15ud}7JIi0hSaas46d*k?^MmO0=rEF{|lfbNZ3LB;7(x)dj zAR7Zx13k=&53n@kzZ`x*eW$W{fqWZ*E#n%uW*&iC!@t$+8MWT^0cJ&INgiB=En`AG zhuvG86Pr^zE>6}vu=o#baOXD)>(mwfJ& zoHVLry!=&_OC8O0Dr&K;4=CpM)61NT{^;$6twxV{X|^TkCn;vaYpt`7G8 zpINnq&xdC&GqetwLYvx$*z%fDA&7H{LlyCaQ-R7|W#yh5V5pD#&$=XtRRtdz%LkX| zKd1w4|AjMf`+o|gBa8koH0!U&0iXZ(f%5-;Iq=u`{{Kq<@5r=T!v^B1d0}?d(c-mC z@by)to?^qVhExR!lnReLT2IA73Dgp!m8*nqZJJuGU#n}XwjBfSBX1AIX1T7O3qGn- zvD*K0Ea=p&wd4(JHl{$SxJsIr`=|l8EXG#)(Hifx$c?Zc51^6Ij*Q5m6hySyf~QU3 zIwhlLOk7U_KR>ah#j~bfm`nmI<{~chlzJVU z#&-eQ|2SL4JUrQy*}XQyNl{PFwwYWo3|{Oqyqv@f?Jq?=Lhj!DdI3vgf=NeJJH#hZ zS21j1A zWG}vLFqS)8l@(4$E7)r*lHpScT-8=pPGzM6!TEPAKtG4iKOuOV3Y3MoG$)eZHnU&k zyXAC=PX8cI!v3BFG7A(N(y|LA4ot3~*Qt^@7B9G<%iVVODnw;USdS z1vL&Tgrpx_?*bT3E@$0;pYUDFP|Zvi*=)A@W% zNc(^R^p}ad^XX(&PLzopf_md2>of4O(X^&UN7=SojcJiJi30D#Tusowt_h7c%!N;) zG7-|RR15UC>@kP3WV6c;zB>26!Q5!{ZKbVjJmuAHEO|ojF){X9OT;D(|7~|><#0BK zI(`+ff#!{AWBtX^pNZf|1;=2grW1$8aJ)Ulf}oQ~zgAIy<>Py7jx?*JFVf6fzK_*K z97s=`oNu$l(*p5K{4)4@_?nnFn*8i4xXD`bQj#aoGoTw}#RO5ne_5_mbn>s^NsL?p zllMtj_b;hgv=b@?CeF$1~oME(2yQ--5&4KiMi4K}XSn<)a!X-I}H@ zV#+P1zka7-;*_e~7p=GL2!Qi5LL=gefv!QX6&@s(khuZtZ?RtVco_$+N3MYXJPgF_ z2H^3d%IopF?H~tJ=E~o;h74h@uAdV>>0_M^U`Ce?Ys^AYUqEvlTsi}_7 z+i+62NJA!DqFx_sQRQxWk9uW>*OAT%0QBfTaoHO>cd6p@bO%2{1JRKHLgaAwvOHU@ z0xHE|#1Y6?(b@L0W!5obb*tUIA(gR<6-(YW%cB26sYONw0F9&uZJZj#*#?*~fC_}9 z#;`|~z)Ikp zt12<%QJ4^_!E>yS8rR@Z_(1%V9BzY_!p~qvz>IThEu+k{BnHUIe7l$)!@Cwvp_Vd2NbL`YX1FNqut{XzwuA; zle_otObJQPOEM1Jd*Rp{Uom9-L_~(NcvHq{+Bb``H2}XnGL@mWr~Qy{OVSaWEF`j* zp;9RA(uc#GIzqh~z!2L}P*Z-V`iifzHSiC>N4_CXXw_h@IOZ_{e*|ckX+;CHgAQt1 z?dv2bh^OP+Bv_2eZ_1oB$0ssx>#X||FUpKcwTl`j{?$AlN=wWY3$W;VcHj&OJXM;n zdKWn#AcoL9o1XUxy>kDcRjldz^81u|br-Ti1Pf=0%3i1TbKE`GWn-y%V1`R`f@zW> za^KC>uhS7FCU|v1dGLm$^uPwO$@cNk^_eewfmSw1Q!`%#v8eC6+CvttQ4}lV%>o(PVTA!9I_aM1BVdeLd-Q(ZV6WW5wZ&Co$;j zP@xQsaPhF(z_4{|w@*x;LKXj6AtABz_KBL#;xS`845KZ;URr9-#L{tK(M$0vql>L# zX4PoGcdPAv_Vi=SMF@Rz}DE_nDU?3PvVe?RrFOx4)kwJfL=uvVNV|YD&)EmIva8^v& zIkYz5{#!Vbjy@}|Fst}b+lRx@XxWlhf)j>gu`rpgllGiqJI7Qf>6q{FA6d4eN2p+h z?C7|JkyF0*QC4(zifAZ)Xu7CfUE!~NTjQi^5|8HvGHI}N6p(B~ zJ(}-OVB>9)YEfsr(*bv~N*Ulf#tp`N6dvJZftj6JJS9y&0ut%r1RFwtAU-ZyTgsH& zoAd?V+f;hAU9Y_9_C=pc_V4|8!-)2|uY8s7wC*!eI4cu*&V;>3RRZ274RgsC^U9Av zpoQ_D&T@&LL2nZdh!KNpb#5)#TLRPX3Q|Q33KNfOxvBt zTM>Ho8jB0b{i?rYoaD?nc{8FBwIwXZV3h#B4_J;5!P-b{mU@A{dBAjno?Z&xH)sJd zHDVxK4~3aqafT%M*r$8LRp21U`;;D{_k%19uqqk8v)I@NXf~I+i2=aPY|o|Nt#Pr+ zZi=pYuu1wg?RIx$;fK!qgS!_^69Y|tA3)MZ*9K=$)b7JuJb_l&P~?_m^H3nn)L8sv z6PS>HodPwQ2N5`R;P5Ko?GS;y=|l*PuD23B##q70Wr!Z&kc-Utfbins1wZ0LIhu>D?h2#oSF&^$t3nXEntx#kQS$o&wn{86LM z`iqZuXgX<>vMSOYDZ@e|%vo$?^f=rBE(Va5zK>;tItLs>fR}pGt!)tSdaPzYcw`s? zlOFE(zr&(hjP`EkMPucpW$}1g0kDTJ9py$%hftss-2fEyDS#WQbMcr4o=Hl*w-a5) zfby~RYAhcft`)(cq=TL~d4XHXNu`~s@8Qx^*xZabcX`2jCH=evSHW`6^B*`kjt4IRYK7S|aCIMTmq zs#yypUK&3bNw}{&h>GaNwH1kZzqg zRm}w8y2diP^v_ik`QCXXXe&8OlJOe4;BK7c1BHTO42ChoM#-d22b<2^_hf_>M3RTl zbR5B)w|AaA!3z!*3+O0|k!HL4`kKihT45p^}4cKXiO-*Bt1ngGkBsB%^~^Z%`)jYK0Y^VqCza9EP!J4vnEsW9D47X2xXWh zsVgQ|C$`5c;Dcv2dK5-T|K&)(*TtBa;q=|a_y>tsG>7q{0K((n_&-{WdNoW*w(uEy ze!w-aG#3(^3pJpe?!fzk1*Or29k|(3CxS)YI`G=&kZpNGGVh+*zsbhImakR9J@Jqh z=4wEq5T8Tj+V_px*O-CFkEjP4Xq`N0bWww#1`?D3eeroGMPmwl0{{4q^urI|tUCbV z9f&khS90D|>CKlgv!#8Syg73!R`KJm6g-d5!o{L<<82$fs&jVc#SW{*dhGl51ar{j zwb|9XoUH;I}WZQgW2Z1!U3Lu^$j+%L!QzS43Ny*Asfx2;?eZx=9yjj6tf#nZ*9J#q=|CvXXVR;L~*u&@)Dk5BJ$<|`Ke zSbJx@;u&dqj@gMxM=!bvz9~ zpiezB*HV4}*_d8qSdfuer||)P?uXo{K&Pc8Gf;cU=mALOdotgrFRh~F`w!K_#WoqE zp`Ot;G8+*2!F+U^%j^;Q&|vx&4Kp=uhGS)XdKb0rO{7u_?&fnVQ7sEa@s!`o_q+47 zA_v$^_G9L+Q;{pink>D$G3@?&jE|7f3@+lvLPCY`TBhEU(yYBm95Dv>5Ahe&Nt&Sw zfRnYm;A`@fJa3_CJ;C|&02uYFIs(z-dOLq(3Xzo*uzKG-?9khxpKbh%QJ`I9<4yH( znPICY8PI3{qQ?f)?*-Z8iiPrK746dim}!vJYk}>?dNKww{_znz<_b~FDORs+OU5+c zb%1TxC!tk<$dMN4z4GGn&9}92XU5tasbc*4GO4zvlP#6*+XnX52oLXi`f7_%mhsLD z0M?mm19DzIF(1L8T{eyxzoJpTC(|T>p9R#FEk8A2O$%VGfDL-tY&`vLT=&Sx9O9#Z zO-$XtX*MBip%$dKhOxirjTy>m%|U7l7MuVS%)tky%x~23uC4s+MMDS>sU}m#rz4uW zEymd7FLdeYIC)1XkfakZk2FJBz7ZsQ*J4I2vI}|nOpD-7@o~Y&o>Gy=)E$m9Ab_IQB1jbQ-oKi}KTNaFDgx9#Kk{c=gCSNva0I@Oq z{C67y@*D^GlNmRc6)AF`LvTFV^6*S``fr5vIr8V7FQ*E336G@y7UOUD`*Y&|UkG`E zu)j}6>Y07t{#RCnC*F}|hRj~{|QS$|V(+azx%`(|9V z3$l~&OodS{2Gtbp{?Ja_y8Y0*L)Cwg7tc}+kCGTOJ)^s&417PLH};=hT}az|a5j$! zwnBdj$t@bg0J6XJd7}Q^T?Vp3rc^(@Ct0*v!+qr;4Bm6O5&5@@_j+tNGbDA@KlOFR z-m0(lW7xFL03EO~eR5Nvp5ZR^v0Syu@Dl4vPMy<-S+PUjqqed|DLPUngw_VX`&UYq zRkxd>5JrAdo6zRWBvJvl=1TMxZsJ z=9;(E+sZiV5FpV_>Y8qXCy@xm)tJ~AIJ#rK(~da5FE+)6nDJT5eYj=KxOtv2*EcsZ z0bBS0+C`VV&}tGZGtK2c?ye1%Tx#ziUp$WqL4E1Y*mLLhBLSKyYH^7}Uk8)^&kWcU zD*o+X`}*@RYWLHAbQnFq&f{!N;LK%%A||4v88K{uANlyP4$9rYNG^30nmry;dh1(@1=t#1W$>s*gZ?Qs@H zo%|wx2pWS=aaxItHcKQvf<0;2#epq?t1MrWtlUV_rStDF%mOvnSS2J55$g%`oT*95 zMx|!?K+&499^#2+&K9JfQFShiQ?3854W>1E0=2AQa`n9`h8>5!1kNVg3?D}r%ruDAgFRGxV@sSn(<+|8 zQXI^SOrFMUqgk=Kh-lPuPix;&z5KJSQDojf$PWktid)q5ST5=jgJlqWY>Fz7dwRsA zAJZF4d+b=QsT+aWvalRf!}>73VcKc+m;E18C%jJiLEN1)Qdi05aQuhHnF1b$omdro zbLyor%LY_SDbjfwmbvxSCaJ2z!U_|UF={!Y{9sDKboz1cTzYvO4f(N_A1t3(2B@Yg zw}xNz9IWd4RlY|pE;)*Q|9r>}3PmJ|pO*-WfNY$Z6LEtF(G-8-K3j>5Lrw&0D- zD)SSNuai6hX-a0E3ed+y<{2>>;Dv{y-_dc*+_~tRM)@YfMw2JX6zyBMhd#}P(2@!2 zrH%NavScKl_soAb zYS|dbA%sSw>er6m>0}}+c^KHLnrrK4yl7mr|Ew$dK&C^S{r!N@hP78$tny{p(jH+6 zJNxLhM=wxSKd}neNuC8Uyji-iVC9nzMoNY=LMj--?JFgQwyJ~qT5mz(gV9RKttiei zY;8WOqt7XbiG~!AGPOCmGg`Cn;kqw67~Djzg3t~YblqLR#yp)9CaVjkz5Nu%uYL{E zO6o2f8^V*Rq+W@GG@~N1S)M;TEmSqIcp!>D5B?(xaPk|M^WZzZ+aGFw<4`m|rr6fm z?nbNtT&z8spx+D^bS$t+PolOdeCboZB{R6v!KpQNUXbaOb|BSbi`lp=z^yZXJQ1xH zpi6a`l9R0;t{uVRtbnL4;d8d+Eou15QciMpK*3)c| zr4l_lTX6Nd#MH9({rT*5?snWUM!dTl7>}GkM1)6m|BM$TW;|6h(BL~;YEIVnsM-IV z(}234;O+*pvks~!2KIhJNGK2E$Sc($p*}C04{<;B2P^Bj#|PQ$SYUo>9w35pYBBMb%d`!W}a_5(Bi$j)%~hfCFk_0@$+<6Fi6kQuBl#| z6eOuOAJ!Fcx~-A7eCTL;eO}kBiVhOTWG+mlrg?XQHFyMF4|c!67+zuDuP2nBJx#2- z^VHW^UI)wtJ?cx3e5xfAF1zxY%u}1Kik7^i5>94=AQtVpyMxB9okn>Wqg|-nja9KL z{Axtje~eJ(`DhfRi73S%V1KOUK(Et2u$4(V@i)pX2GGL1ozOno^I{@KJ!_ev%IQF2 zaBT);Ue)vK6^daWPdmUeT(0*FTJ?uFuFp6CI{}$HAgyda0?#TPWWUtWtI-&ms2X5m zy*eO7{F`bMbkclr8)}`l^!gSTbe&hwt_)BNsr%psq^Q&?lOZt6zD>J(QML$6M~mSw zT>71U`N8EiAa3b=*L^!DYU8K41%bPGK>_gl4)W5pPz`FsB`Sw8eP6li?eJI0d2tw` z7co312y~zcQwK&YAO}p8kaf2Ky@{4%k2)9RJuNcQ{gK9*{020^@}$Zdq-Y6|MUKx* z+00?1=W6ahu2V~X^yqXiT+;=0g!ae%z4=FP1L=EF{qt!Ip(;(Ss25nF%>^dBUR zCA=?ylJz*xtH7$cYfm`E-gSOPu?iS+^jNjH^_C(UH_}+qXKQF@Y+rS}8hG*6yAKse zbe5#(UdhQVb3Mpyc?AsVXBBJ;@~b3YYlix`6JUZG$Tb7MEpCL_UgbW$r~afT z)RM1uet4wl5TOWFBg6I7OR3!#cEk{R)SpuChm*{qP^TAc@0KvuP6btZMMI`}O4|zg?QQk)X(J%NjIGZ+k6CM z#=F~{#ny&Xd+x1Xd#9JMcVJGpU75nh4BPLLF2PJ(iP7rw+j()F?&ot2BbED5(g2OK z1&{_do%;x6J5F}MFLY;E5L?3-aQ0fXKVIstVqpsF*svK9NaAL762D+gTnLfzW&p+d> zk5?n;{A@r1XYFNU$z7rb^F@`SslrV`A@u;Lq(BI@&DlRrlXGjUdjZ(g` zp>2${TQKA0gP|4AtIDtj(&%r3>EAPD`3xUx-g7Hh5HW7*;M$|HjO)s023^yDlw(iQdh8Tss!ZLsFI%! z{*Aejzrw|1Ybpm32BXB{yV@jUpnF_y6wsQmb;NULD$DzF57_ZHcBh|++$o-@Ho zz;LnZLq-Pthmcikvr(SWJunj445*LSP;u30pO63Tfxb1kNap zTHzub=d)k#aBiz_sqV6;_-t66d8U9CH>^ME)np)inuyEf)eJDJ=$_)#vy*=9iyqv% zWq|qD$#xRG`EfDr#LVaOlTA#e_595i`3m*9F10%FAU5~fB}S^`X{q@A_$lD(ImCA^ zH28=Tx00&v{ESFR!}+ZFt0f~2aDFX?JV%%5n8UhRTQH~m+_fXDiOu8+)%G#~1B=uh z+EnAIG@YAdb0d^Ag=~nr-)`YmR zHwlGcFaW58ruuLV>OW0s%bfiqrs zK>$ZX^tac#5!lzQ<6;pPJ&Gv=sC+ft9?nI{&*N{HTRgLjXmEoHm(38y^~0?d3f91D zV@<#J)c{w$sA1_|>kf!bAG*AkqP_W~ke}Y6EIQBPzZd1MB_ZrP$dIpswOL7~mPG-s z=U*CA@yt@qgmy6d*5jR5Zc7oWbIe|vdw4&jY#Vj6gsck6@B*L()=Nd| zrRrithD+F$&f8%?&Xa1c(gKB<1S747%O^?v8ej=1BQXo5*9SK{Go^VRt?_5GJH8wl ztZZ}_{R76_dc86oOm@q9oYK|DT``l4n}OAu1D?Le%TjcKMu}a!k=@*&T34|6d^JoZ znf?;LUM;mhi2S}f`K!z8$*Suyt_|LXg|yP1sje~V507IaNw8F1H6LX7esNt4{&jmP z;gLzBdC836?{`ZSKWrtGzKCRLhOCE94iet`csG#C_tVf~a9Sm8rpO7E>CaP2Y;botz>Nw zS*9$b=sy$IiS6mSY^JA8$?eU^PyTeeuDI_T@tUsKPX&K`oGx(Ia5ObKeAroAa($>O zz1v6?PsR-|!1YJ_9%Wf~y`jpmd+Kfp-|JAX$dAaegl$i8Ah;>rFUhacuS2g(i&^g^ z(VX`5rc^mNlzFP_TL?5!C_!?6e1m zKU6SZ+lcjOlWebxchASWgBl$MpXI1}crH%7zS>Y!SV-xmDSu&XVsu|bVJ|LWF@87& zdtw^po$*Zz+syZw%zs4L6$%_&?rn`%dneQs^wN6%Vjy#KF3jfAm>W;*Oq^;bYKJK_a7AY<^dYpGHf`Vo2%5=!;mIhYI%`J;M2~ zR=eX(95Kp8mzRYD3ZDj`bTti#P0+GtUNAcBxS1Ocau?+eo@0juVq*55@PRFy%~N;n zxSe16?&={_RL2t%TnbNf-h0eBPo@R+wZV^-)q+8{1Nc1*FA&_u3eT%TvHv-_w4Y#uR6M#SGAQlirq@!aPoDliOL_w%&QI^J+{I;2R5iHWo>e`BW?nvf88#DgS?#TNuXr05=qQ2Sf2127E) zo#xlvK5WiZY}8Kkrm^1@I_}Xt?M>K7cpD7gO7vJLlF<~f$@#*ndA17AIp21i&tdDN}g`c(+0QHEox}$1ZNo9V^Ul@$_&cer`Jd>K^ zujEh0+iRV6r)9L9Sx=)_r+dFSN74L>?7Hj;1|$38mh$a6vV{h({;BZ8uKl?7I`WsP z(L}aozoHJ(OEVu%HAzODPrWa!&2`wiRASFm9Lxgt+Tv9h;c+a}jxlFzctps;$a?2V zSjFsV0hqvV!MJ;0KX5z@nDtuj z8uy(XonX(+Tt>uIgCm^{DD$-^DPep#niuN$U205hu|7h>p!=X(7zIM0OR;z$rM84kYSu=#tdf4c^LEjzN2^ldf)5X z>%Fes-}}$HTn5iP&wbDL`@O%P&-e2@_kDje^`q93-1)o7*C{Tw>y{R$xbA;*X*R}B ziY`m@q*s6m;Qdm6?cucTMT#2xGhR-(+o_=za;ThcK~BnPdjuLcX-VqHAzI34cJ_R7pcE@i2RAuVFaa- z1Luz+)3h&fWyu3dYfxPiJgX;5Hx1nDpdhx)zUrW#o`ItPd-7-SN=ooO2I)%Fxxr<$51kRys=Ds(XtUX6b!8E#_52nbPgr zMe5WyV_d#ggOvig<5Zb;DJJig@d2^7H-w&`hl#koUZB0wv8J%0>vZZ7sLre z>T&G_zWL@zJul}jYK*_sZ&md>Hi;xfw2;fMYE(sL7Dh|+DCpj*Sz7cHY=5S-C3R}JH+b(-xIFTJl>nJqp-WZ7fkTv%ES+d>$}=lw@-bm zjt3(1ajsMO4pJR#;caYe-1MmWMxq5?Jp>Mw9bezBg=|W8)(!PcH`E!Hi z0`*xH)$Vl+w{Z?$K%L~*4N#}|!Ao^u6XJ@)T_}9G0_IzX@TDk@EGZrC>LA(V2 z6=~WBYBi!7AlR|L{AvTC9_Ja**2*jL+6__ROz;~tYC`w<+jvZbwxyww2F}XdQ+9El z^L{O1bG)BM8sz0YdIK7##Sh zBy*rI1L29N<$}IX=b2Q^c?B3Yy9kOjW3@Y_$Fm(z%N0B#&*>**CsDQ?(78HuiGQb6 zA)R4OM5{dS{{oIwvzk#&C=4Nf6mk;aM>0G8u07!iP#J6 z4$q)9pj)F8f*4>oO-u!dUv^iQ^$gG}GWKxZ&JYbf;D$Uo{D?J)@k7>*_2zm;MyYZE zthvULn#`gqx$k{XBracR$j!FfF~DyhX2^v4^VTT!2zsjOzK5Br&dnPP;vIp^ z&68LF-K{la$C>p|YXGHNDZ>JC%C`1hx7Nh|XSMTk7)DOrF|7O)kGyI*8Ow|Ou8CnM zj1ylPJU(UYHTGILT1Lj=+BMBd*ES2!u!W%z#j#`iV8H`<)1!xMflqZAp073!87*y4 zZt8uRJIkr2A(tvfAOE-=I8a*hlr?Dpjabwy9?{a}@+gU&B%ddK%suraBYvyOf?uc+rqPdGjFKFxz8%` z_LEUPHWo#ZJB;1b!l2N|x%(_g5UPz(>YO}Us3JC0D<$86FMOYu=~NqpA0gxLqL0u& z)MO~Dw7}{RZil_iZ~>Bd4)Q+ znAhv&+*U8H^qY;bJ#|4G+rqr+R~e!SnZmgeEIA|GFHntX&JEEBS!~o(r2Uwd)aPVo zuYdF8qY<=!e%|Etjh|B#R6W7JiDwA&Pmt#eJp$yK5M#01FtquwzSZQs^E@bauJWVa@cyGR_aW#LfL$YZxejr)5#}LbmNvpmBmxGk_xUy8` z&iY!ph`VYn8=v^*dtVJ&$Ud^!&5B?&t#`41t}%6a6Tr{Rcy|V}uhRvKIYYk)@<1a? zKGnbT_61r<1Cf+aUszbX5&jnr~gBc2`zYj{l3$e;Y9i4o%-0aSCQ%&NzWYmPE@*0XCOmZSwcl%T_r2kYSW3lw-<{Pc&B|9#M$L0 z5&(9tgt*?Il^#0iuJV?n&3yR_IyiJ)(F+tRA&x4p$i`v5thlleSy`BJ0{<`@Qulag@j)F)Z$3qA9qTDWXg4!C(3bc}Lt>6X2S{GW zErsq;C4-IXU@JbOZ&5;pvWNCMq5l;)q$K;jcS=VfM6S6q?h5x-5N5v;`>dUXA-WE# zr?L#PH%wWTV%wO;)ZT9hZyY*GtM}<7IhB?E?hbCFW90&f@AH<#G*acjD23N_(JJJw zj63!eWl2fXEsIG`v)uWXmuU)GSql@4f$ll!7`Y$U+y$IKdC?iiFE1J;hIu2SWoi8p zRz|1*@nC2bOxeAD$qcWdCy*|De_hw)c7CPH%bSVY_a|<5jm&iWxmQGYs{BLja=kns zOU2tk-R8VAI;~p2trHX0IU(PItte85_&2i5K3CD8@#>kKdt2kW)eCsx%;mnimcA|z z$#MtpOYFBssowqWltpxcJh6?a>;l>luSbtX7?o%P=!<~Z6A+%VkU9Vm?y3(Ls1=jn`j`Q#tI(f$ z$q=_dO|&N>V8Cug*1_(xPhIwu9|BPGm}t3W3StrCfw$UXfrTaj@5`iT(d>=r_#zUf zLo~QAB7oK0Z!T4q;N>v=DxK&XEEM^7cBlH70ikd#LxNImAg0JfFNf#u$f)lAcpRXf z3~SS}aGhoG2i??q8DyXS*9CogGpv@&+K+&s)^*G*59))}EDr9GVcn25S!R|S?BhMo zSE_SD4TWSFV1ozUpTqkKuZq(5il423=amlX$FDNqyj-IMlqCRU`kR3M*K_;!_A<^U zo8pSVBb#!EGZ|gh`*7^bq+|!DsH9yh{k^+m0QlmGz$r7&efK>E{`jkvl0PW}|2UjS z+b&IORxf^OpJ3X8FA!Xv!4>4%b)ca?-$|4;4))GNAK%~pshJLG3=VK(HZfb0Klyunn#=ik#RwwTO literal 0 HcmV?d00001 diff --git a/src/_assets/img/performance/trace-transactions-spans-generic.png b/src/_assets/img/performance/trace-transactions-spans-generic.png new file mode 100644 index 0000000000000000000000000000000000000000..2d05898975fd678f217bdcd64faffca532baef3f GIT binary patch literal 30408 zcmeFZ2T)Ys_b1qh0ul`%Nx)1_k|d)DhzLj$$sjp{DFZ1C2BT1X z@>m@PBPM~t&J&ZI19vP{3zNa+qSF%{7Z{A}3iLxTH0yr?gWZ5BK7Od_`DuL?UaUVJ zIC0?9&!n5lCgo!vhJRQp@-d!EZun0}&7E5lW{j7g2V5}wLC|Dyp``Ztg{;78Ax1j6 zobROhu%3voR!b9)U~|OY6STY%3sqSay<01_6qUe6X5v!b#96 zTKf1a?zjor_9v(Z#|a+_q#)S7Qg`Y*gXmNth6OkSxcM-S+-8*)$Jr8Hn|S3hb8Iy4 zyeZAM3WM=cnot|ZsXvw5p~`|l%K(j5f?XYcfyxetrdYlRp5-sj)QZRAy7SLzEVR(W zV00h&Ncjk|v;?(;vPkVDcg6@{#=c+KTFuCdtdK@p7Ln(MV;Mr3VX%&yk>{TE{u09? z=pn=#bcP0T%|#;|vc9tMao+>0XiW-M3+PgmC89|>Hy{P%kpl8;S6>9fR~|s3-GMMN z>Ei!CAu^y#=})5?@R|UgtMUZG5BJjl>j$8ffc4$=*8vwvZ;mMa@i^J@fd8dZ=K!Xx zs-B}GygDW^9jvRCP*GPhC*y-YXwJP4~g#akWa1r{6L{J*GvGrp#+TfJDBlRq?}rf{RjzR>Yd8VflqA_EmHG>4L09czbH0m6ld~9+&ae9Qotg3Sm+ig|l~8 zzsux!r6b*tP_fD_PfSQ|7ryX#&w|2lR&H8rZxLQ*LW4Od@<%#~5Tq6ssz;XIKQ9vo z9=yu;2|rkidf73^nW_|d&Mvc&=NsQAkw#DPT(>%ue@SItK;MT#_sm8P(zEd&_^0G$ z>5JCBu%%%M^-P{WqGgSZK4qHraGVm>P39-HL!c2KJLm&SFUb?!LwA^0r9?zAyflum zHw(03sH_W6M!zEphz7a-9y_@snZ4Ng^6w6t)(Z3&U^hfi%*3!C$#+QY0yXPn`C)1Q z$qfDjE3&IJ3m{mz4l$vCzP&c$@hCo+!EluV48|7zt1jcTTYU4Rn6H8?>Fsm)d%8z& zE4X@M{O#Jx&kYIzKMl^I;yHkC2f5}mQFCAT#Faqm~-GFSYZHR@M61?zOq&`o1zBQ7@lCRlR>-hjbVj- z7?SE5fr~i1o|Jj-H1+$5pHI4F?+QtKm9IDx=dYE_-yNELON;$GqsZ5Wg76nOY|wji zZfmD8DSU&^Rj%;Dx8^%Xwe~HBWg>RPu)5pTbvaaWZ-XxEwR>gNRj(UZsy=OX&3-kz ztlfJJX!m}o07EaGc~DBB7MFp7bbhARhgk}4+oD2hg+**iXx^bcqy4qv#3f+?`9MX6 zxi{8kLb!T+*Td2(`m9#!lC_iYWDqaUT7N|mtO^_!7M}E)wZCZYuu79pO$8+orQTu)aRb9hZYF|N_A58&bHgunw4AWChGy7|-8{NT;u?_mxxr%fV(YgDw#&7}uMf%L(*#5^=_j!B z#vcG24Z0a#x_^zBl+UA?o#dI$YxWpX*8l1C!QkbtXpkW$?t{GPDtk&O4wAWX&n~%8 zYjC`H5$0Iqw8fj~GkZWWPw>7`$fZ?edz!RkIe4Nr66ALNFBc8|l=_XkpL*at@SB=< zi?Vc-8RcY)sZw5c7?zz3?x)pVFR{%y!v3tkGOOd;f06$iowqst=1I~+0$9Lb$b+si z*q1!fBJD`j3ZA&m6hYiup1@OgxEbNUDKJ`tw93jCPHDiFEA~~W(U@PBos4k4c>q1T z2Z8DxavJ`QT%GL3C%IR!FPwQyIYDzq#P>lA1!T#ha;yoo}fVl+-K`IkizuXp& ze~BE8PBvk*Pw`y60v3ZSU9SE7K{{Vy(6T{Ty=NB0mdgsm39 zIRG!rUmGtqYQmn~S)td0!Gs4+Pkc^iR?zF?___HmKazTTaSvYK1*GriLUqYN!MlW{ zcIlTMNk;I6zD)3NKrU{?%-feQr-CxQ^*6=hcWJM^`h~t^zHW>anOqiLFo*ilpnRb6 zkkqmu3$d!y?SV(VtRN!=T(qqUaewH;=c(0&d7_+TRcM8Jp{77x?saWC|4s@SG(?f% z1!{z^mQBa*IUUEN$cp?FVH?3fWg*Xg>nW>avH|g(XP_xy{7SyaYLu@w*;Xk*xZAs? zQ;Ebajm=bt8b|$fvj(R3IW5Pqqp~Z@DJ-0)m3Of$Wn&!HL;=WKBJx znKe$w-Qzu!FDy0>bs+9pm!}{$GP~5htJQ>!JwN2oMYs>;1CEljfcX=qH9}SCbr%2> zpNSjNY3D9OAseH>ki0GB9pi|a0qMe(8IW{f6I_jKRs=ObN1LRq%7m%8x@^Szpo}!P*dX zg8B#1g1iV}^X4NxnhP+or_i)~7-=3ayQvuTMD#N;VMB zFi>?VClJ|JzewRqDWG%crK3?G2%!AOlTxRwbn?{NQ@=cFz*j%Ryk;eL3U_C){rykNkvb zC%$nND$@opvgf!9%pK;P7D?VWIPlv@|HL8%ONL@M;E!$5$}0x*?l?&T86rp<#@(S> zfhKp7O<9EMv{P1$ z-DM{$+f@Dsu9otllP&}EjR$i^4dc;-2`<{8euiFK4R0|r*qmn47+-(S?)dE$zsS3! zi%!%8oRAuTB<)s(h5qxAs6VORav*98MY+WI*?LQ{PNY+8<)5Gy-m}TiZ$=FSb)L>& z)+l~*TdVp=iF0{ByX_%Owwm!pU}#tpLQJpzNZrHLxWh_|f=A5u3V&UJ)F<^(i8@bj z0tY}CU@)f4h#{M4E9AUtyy=_iA;jiT61wl(Kh_9Pd2-LCA|JB|(7ST@4{$((y}=j^ju+b9ieLh!5vM{(5qB7k;QcEx;e*Gqkl&M+gRKMh9NAfRGK~He;8RaVOFe zr0GIz#uLd=_ zV|Fd}30xXVWr4!|qKumFew9z@!7`|I%vOc88cshO7!ozC{CNDG3D}>(+V*xTi<&ckz3bWP9IXM8urH{ zd%?}Zl-!(aEsCR1(@HOrvwDDC`8sCgj@4(4v+zr~Hm*Q5Wi|V?`+fpCLPddG#3E8f z*NVVUZ#E)P$QR&qhaztb9aK zbZbQs9o`*Q$*T?idrR`-d=jn(J`~aJDJ#FOuRNzNRU_kq(hn?+&pW5*5%S0M6ULG< zzA2HQBno=OcxL#yvv`KBd;-M9@?`Si45b$J%lD9sGPEIb7MEPaHPwuT#vlj%@3i0@ z5P+P)ey?=Ll=?bh2%FMrW-At>G14B170Hfg5f_{Iag}O0`gDu+6g@w^SGte6p-RX~1%U%2xqD0H9&)~l{ z>*J1@tLi}nlzc43-SOShkey@tGj*DHdsv0!-x^*4QHk2<01`#t$`mTV>I`ssD)`WA*N%lc^UV&&A=@+~uiN|ko zN)VKpqB{nl1myTcQq}m8NK2D3&Gajq-tU>ooq11uyo=>P**>OJAL0G$umHSW(rK$z zLxFj^4qj$Jn3~P)ET^VFzpq{P-Dh|!>UD@~kD&PH$;)53Ov`A!TkiX>EGA2M128QZ z@SEnC+C0!GTqhlSuEpN*h~{jXg}w1PY{yPna35BtV`EOAwS(iJFx0U1tNbNWu0 zH1%fa^HH+8h)WSzxZG&1TJ1{`xk&zR?o2IDNMYqP*EX9!s%z1V4OKPn|I9by6jjSG z`pTBNkwa@_fal)F%pbS55sK4pI9x*^UBI$YKKwO^onJ)VqmDkF#Q590_;KKAWxCUD0}?`{PZ`E?SfgzRJqJX2terDPJv zqPKQ{^VJxYw-n(1d~l4xvmUUVoO}smx$eS`Z>U;U&5+o^LHE)9Axy{n^DaHSmA~%Q zm>leO)hQ60%;Q_an57<|B@cKF>s^87?xybkk!;$q6}ATAIwMFTb@dgD>QIHr{9@ZX zh96Sj@bzXzXlY&tS!#j-cpV1BfUQ}ZBl2Y3iq7EW7#=u9)$h!lJpAE70&hDDu02Nj z?~g@QU+bvI348_QoQIC8*QNJ{BFdz^55MHaNYwtK)p7pNpC)-4wQQt=7Xl)?tgoa3 zm2Cmf(l`CUV_3fnhNnO9xA_h~_WnEOvD&$p)0yFER#OHmqOV2W59OeAh;pdT<@xp* zYI57%g>?4E{35lV3O!%Y>J9eb}4K7x_ z%DR7JK-}*oXu0l2Ncin&>6kpbNuJwRelGos<2*bz8Rj%I02wP`NJM za3>O&`1NjL-vxEY#{+)H2MF*TV)s{3R-F&9D$KYA2xH)r^+o7KVg`6k64|tMcDrb* zDT8usNv#ro)=2U`HWh*uimU^G2_x@2Ly>mB)`yxNVe}`QPumD-jeKlfZ96CPqwI&a z%aAS$8%stfAi#lMe=fcEDp<4wd(<$PEO6`vR6Erye5-X+;9dVX$5HZe<{46fZ)-sxT{TJBG0+3yC((V-KF@EKs0hl}=)schf+nJfb86n_3L3lkK8qmr*nRN=k`HBW=MHieAl}|kp@Bn`0d&> zuXNM?wNieM?O~C5n*vN^5x0c2^U26}vP1-Y8v|sYMJw+(T8b?vQn4_kEr~S~k0|v~ zkMF-K#%Sp{T9QE};eFP~`%LyX?X@dKL@eRQ-~MX!EQdis2hxY`=3RyHC7`iR^N42o zf1d@2MpT>_ZfRUk9_8us20GeTC&akH5Wa_BmJTLC-vVZQJz+?|4>OyL-!j540++4z zjmQ1jD>Gx6Un^_bTQZV-4Pe7o6IH?+1H66(3h$cbU}4FFzWKwwdENUh&|ZFka}xj4 zFbVv(+sFHxxW2MxzDC7V5Bok4wYEp!dVxnmR^O1!m;g==4D7x+=>}fQ+j6*iEoKOT z^FiXatAJL4M$CLB18Ght;(N4_e@{5YY2B}b@Z_F*1kI9sFS7C-NW$2b_eYUrWa17D zZ1_O2r*p1)`_s)c%U9a}WI*=p?`x=PIke6zot-pv`iwi-?Gk8LY5X120=ypLD6QVJ zdw;$AUd6|~H)RWbPAx~Wu%Wsx$l;<&XMWXh6XdFanooxpWXJ0>hPFc==ocb?$K^u4 z)88{t>e+PSVtKmvxXvSp`GMz8h+M#by$A8w48;@`1l<4>oW<_}dHF-``TMO#G|e}8 zm$0eZOj)Tv5x92E$4V`@GP1?c@p#q)EyYIf6s0DN`dZoVxxT_^JHNOB#8t|DJ6Czo zf}eUfiXR10op{pk7jcy@@pr#0m2~|pnijzD84mg~!tcfSTX%WPHIo4f1M4jsq>FlG zwwBx8db(6)(!rLxf;U&*0TQXgmqKU}N24^GR<79J$(G3dlDkN+)3$-KmXLo%fW*$s z2POa036x0U@2>zRDA$(bq?)GQ;J6WbIFR!cmM54&K!;zo)3&PzlSO9red!x%z18tt zi$5nj6vfQQUkj&=P`C)XE9O@Laa#ki&fQ>-1<#&tlrf8fL<>@7mCryah-Un3h*Bc2 zk-^{9WYf79D|`2hr10Mor#N2&B*XiZrM>iT+UI4F5OE5^crWlk;Zz~VMqxes#4~%5-$RawPj5a z+Ve1K-ddy)@u~7V+LBc&6&`dL%?hKcS#cWYrWd#L?evHxam+ z_v#N*rKuHw4$(}0E?w|?fGpfiZNh)CN4+Zo4@$WGuHc>&154M@PAq-- zjpK#6T0%?Z_*zY!{?md=7XjKw`s7)Audr>sYWA+BK!`?QU;pJLOMtzmUM~85AR_V8 z%jwr$ybc8hB)uR%tnLAVrb}&4Y>UQ2WFx+3v$#bILV&?`V)zaxuid2c>y-s!C~%QG zC!CAD+Or_G!Q4Nv#5U!&YtL9RK9p!~llz{iF0bEE<444Ea)&l*M}!Eiiz1yl zXLS_=i==P9&KBZp&0{@iEr3@O5=%?>KSrTWS0S(gR@X;#G;^l{DRM@~7Nx;zkE#6e z9x(vg#&$3thd@Yp0z;q_3wk)sd~!w>SSA|EO^+~#VNihPq>A_-)k1t-8fRqa3&;ur z(3jB;8^F?xq{NhDIf$AG^%qhOmmI`vWA@jlksb%>2JMCo zz~QA&5>Gv&%A9(n`Jibb8rO030a7N2ks4Ds0L-;VwU^yLbwaX)rwDtSey@J`O6aU> znEb}wV#Q)c33QKI7>kPRFJgYS%!ZjlBeS{sFOWNk_1kEot|#$B5b%FJOywK}`U~jE z?_|>WIcd@NJ3RPpxUbbgx~i+OEDY8Nni()-l^jj#?@iJ^o2S@ec#^C3_l>feXXcGf}H6GsO7r+>AxzB+2V&kJjPeOAH#kDo9D=L&$x z6AJzg3yvq-ye)7{P$pX;x%+O^J1YJTx>qDD_^nxu*;}xH=(C=p>dj!WNPn~C`NLEs zD14eQrq9^>{%$z1yxvi~Y^t;NUA%6dAA>muTe$_bN4>7SVhiW<4QO|LuzBAr2?k?? zcEI@R9!a|0|3;U>oU;8RWUswg8%4z$gC2TA02_l&(izWHvzPO{;s2J|pUIDQ8%DXa z`?0k&HomY|BwCRbh|HCruTmQ@OV2~mb4qc*1l?zg{8MA7+Nil*Js6DZ zY^~zA?W^tllNYkPQm>(kGGVYDsNve1Wx`mpW;Nzv;gPHc3!nj8V6fhT?>9V$!+|QT z9=Ag_g8c>^GU!@q$}S0WxnrQTd3qEn^{MdiGocbr-_TBRfF+WrDQCQm+Cj2Wc9ee8vk~`vxo$N!?Ji7b6QLaO?J#J9 zuCCW+>$N_F1$7!iV2wzXgdHb+x_??DkY_b5%uCY=cE@pu3@d~9a1O1Cw|%NVL~-l| zD91N#cgR__UAQ|rFA?%trql~3@r$*`aG6(cKuzdAdS7B;8_H}m0w#g}$J;7@&F&-G ze5!F+GqV@9qT)gAUgFU^x)5HNP#<3KCS)O zG~+ehla5NP0)r?^RT(4%53?;k|epQwYJ}gta)`KdyIwT)JmGZa(G^A z1GJf^ZRR!?@6dSV1+lQzKDSC(m*4dygLaN7*|mUy1m?iDDn@3u(u;`8#tW> zL%&IM5Xj%I^r;wnccJE>Ce6uBUK2n|42o72t(EX0K~ zzRbPVWXY*uE6iwG&JvgMIyl~Jkv@Ev`$x6?Cw&(FGJQ8)-R$2lX)H+bCN;M3S>#DF z*huv_`B<+Jy1ZNwX)fs@HMfUqdd>2Y9Fj4n{7_6RI97f2(6;G))&BSSVa0bj^7*vXMa& zv{`!J9cs=p17D2A-j)a#mlSarnzV$!dij zW6UfV@osJ0nZ25~#B!0pMzV-2;`>E0`m|a~(kJ;w)ztjCgDNy`V+e=7wJkn|2@`jr zfs6d#N4QM5FkJ;C=CgafbWHOL2(zul7C>~(xD4!Z5Znc;1Aw(W8TAHC)L z4l)cUb8{>%v z3cb9erzJOe2Vyc zpS<2>&Lkg!R+!?l!+)~LP!4@xlqQ+n9;p_E7oHgzPR)>;OXvyYBW)#T{Dk1DMR^aI zlUwKVD2}TbTg{E~ZhKu6TZ5jE-2=Xb~zer1eJ9Z-aTq}7n}4>1RZt16Cjbrv=t+_ zY5uSE_?kA^AM1BltQeK~HQV}DL>xR>v}l5KM0_nsDAAPVWF107@)4Pcvd_n%Zk?Z< z%%95%VY{qpd1sPL*Snk$klLET#)6}SSOgX8Kvn7cNfWuI#a_#oWeb-DlI+*MrA>O6 z;%XPY3L{UkT*3uyHOiT&wb;lxpRpCzZ+iVY0C19sBTSq_$4!>KhDOtEbHA^VJor0P z{u2#@4c8oUt%u?bK8!P?R#>}F6VEoz0du$z#XQztZ398rzYYS6cx_G*6P*u$FdVx5 zztbNR%>2d@k5iDqK%W61`p zc74jF*mE-3@GHdN9H0hVB2m_3z!Hvk#CP+$8OtW6?B8&d?ynW~Bx%h(s~`g0lEO4y zISc$hzE(#N`M?b+35W&C!C^qqLeg*lst#B0v$roO^h)Obfdxlc^@RYzjk&^ z8kbC!g&|)-Te-^+DnWD+wl=6DQ= zDE@rv4<8H5S#9Dw|F3uahrfLJc~47={u1n;xBho>@!3rO@y}t(HTiC}Hm^-W#JN@Z z(X8C&$;!U!>GpuO9EZ9ZHG58Jj9S&`7ZN!xaLQ&q-eb;9fw-1zh3%ikaMXN|mt|8g zlva`>naz+(ArdfvB~S;PkbRf zk=9G?(t^B+ zHn#cmu`OK<{W5n%>dr<1dq+TWkQ_J8eXK1%lf)2tdHOH9F_PklZsh?vol`ibgU*iv68cy)YK zkRa71A3e` zO^n1}TiNo9IeB=P-Hvi_2(8^jD`AFRdEPjRg!} z-|m$U(wQa0?Z*SG+>Q4@}-Zf6CFB7#tG3Y7sLgv20xFD3G1l$`srHfY!zg} zf5;HMfkMLWLcA-}k;&iBJ6D^?Z5&xN>(f@Uq?F}|;bT>X+2>mW?mb#QW|V=aYIFk1O*RsD4Lgpx9{;x56BtB?w1 zUPvXsM-SV%Iw^c#X{9$7+_fIHdOx=dBe(lU41=U7_uU69#$g-7cU)sNsRIbklsS08 zM3^Bvi*RpHQI3Fa>pT>T23Lba2|X{Vd|`t*^BJ=^o+|zt`+w*0W2>J|F=Ztzb?{HbjB$kDXT!~vo)W<~ zTy$m&t7~g(>go>m4ErcYl zGLX&M+V2VNPldMBQTt;ZnPX^)iC71Ac1FS9w@&SdYN!}?osqMr2kC9!-(^-h6WW(e zosM;KUU%)vk<@?W^=oJ8-N3Sy%V!1o80NoqYma~93Ou)dDm|4>q|#HrpKbPIaDa-3 zJ?ZN*GM*ebj8k{Do>Dk5?ZFL5L4TFPNy@pE306uusp+4P~$r=KNxKl3AY zBf5RGhsrO4mz}5Ny1K6-_FhhPeJtM1t;;C&6g!?9s6q68e_qv*lbt4o!ndCvHYgPI zl$z$;poduD94RX+FVAQDoIz+jtlwAG9=A6ez&Arv zXzqs&@V!fx|Mo#5G#!M=Fc|&+bT9mW?Z*H6z98*N-gTfeHhso1$BT7-Z3a6WVfQ{g zyTN#-*0%-N#tzK1JKlPE%@I>3ik~a^dE;5K`(81xp|;=JC-&h|p11AWRW(QwWcq8& zXcbz2ye##@T#@+v;cRW~7)?rK1~}o7<2e zl*v#-!Km5#Xvrm}>oEGb!K-_2QE3>Ry%Luf625T!X^RnjRJ*+_W2Rng{taqzGffwa zDckV9-R^UZSEu|CaUtc!iYbRY0UI3$19oU&4F9V&gDQ;E*Xc+y*CyPifyjwmy2&^FDD7S% zmqrukUm~63Whwj=zq7$Y9?HoYiItyji6F2sD4G}ZV#`J>M(JUxq(`bdqtn90^H(VB z1(NP#R^dlRO5d|CjquIkm*8)i-~Bv#Xm0+i72gg?K4sIF;xhYzQG0pEM7;deN!CC^ zO-=3l;C3^Xv8>GYq+=YZzy?Y>i43c|>6H5h3?c&Jn@ zkUN{kVZOEsQHrVvDIM}%ivR3ji|f#;RGZi?qRs5Q@yA-h&tMU#U zjvW*A@|oD)kMZsi;C-xty5Q00<1{IW8p8wmq_N9x3;9_YzpdobyC zT6ieJV~1k?Nr?kAJ|dCp#{fm>N%qkyBPtvP?5gXud?E1bU(-Cp*&}(ur0200FMdZ||0w2W?VCgL;!o1UF6*7!%pNS&Wl2BCo( z?0oZU&^Rq#_Ughoa3$)|B!bZ4JFb;K^|8SPCB@Y24+jGO`w66elx6ntac~ zk6(GF!338;q3WRMoun%&SOHwB0R9eTNk_%gWC>!zzjmJ%RTMowe-s0~qkQ_QrL6lM zZq}D-yU81Cn2T5Dd6wFlXBW+R*8>nSgt#+$plAN%7EpTu*3A{iw(x$z0QrcMrY0tBa~8r?NGxUt9f3vA${>lSY|%bsf(O zKVB?P_I6ulL}|K?eOg2CEn_n?Go>-Ul6GYP0KN&0tE{UFxb;=POT*=Z&^Vegg%R-q zj&r0U*UqZ6))yYSGW4~q$T4l$xyWU?nU8K3VV`GAU72@;HM8~`@-^Qd%ivu};b0B9 zFu|2<_wTa+9(~i>y<`g+_I?JYou)~9dL4%R9B&mY`x;8C(2!PxkBotxU3l?&hyB23 zNEGgY7cT?w{PBE&c?225bTFy$;pZ1SPoYYHRdMMsx#JqMVX4Ol{MOkc5G)VZjrrCd zt2$gH*7ftoIS%2wf88a^9EkF-MbjEEl)q$W`~9U&g%sw+LCse5{e+&ufN{rf6^3mo#B3LQ~YK&#=Ta##L5%II%GZX ztCiRI8l^UZ5w5NzB>D4!34#? zXS$+_>Lhl)f=K%$mie8PmWYap68}02^K7?oTt9V2vboAb{sQ5DvWE_Lv`@5u0 z{+2Z5^Jk4GTXo5rPyA0>G$1^tRMa;br}Jws;zeFozVg9U_`E_WN?5v!lLhc8@?CWiMz)T6kS+u_veS^CZW{bkl7cK!ir zWk)1qG8UcftOV|_6HxJM19jI5(8%@-<10KOrz%d;9ewv$+O=(B(xNFJGXIbu-2!>4 z$)gS8GbRcN^Y%WXQ!aOV)TGoeOv6!ieh^S+RRc(G)Bl*(HM9SMoIA$f2beRh1rG9b z@G)a=?RXT^`NoHQq%s5=sMwZX;yYr6#<5?`C|}i{tcI*SE|t-?MpPz#=;T8=D+QHn z@jd<2(zp}l^r{-*VWQ!&d7?ZORYp8fBi8sHL|-1=3iwLn{<9wM&YPry@KXWw-zW zeYBguIc6P*7Iw{nZm2f%bcY&jUtwHTl-m1z8s=hJ*u=je8WQqU-H&H+(5 z{(O9d*f#JDA8@LL4EW%=<{G3$TiQ;p{&CmsPlRx(>)e$pPhm*^@l*1$>wxTShlq}c z!M{VAEVSHc z6?Nnn@j`Q6%o1cu;Fj0@MyB{*SZFhR&?MVPNgR1%!NaH86ny4+gE_N(*-VjR+r$OV zgO&F4wE_&U0M`D@WXDW!%9nT^;UT=3e)spo)ofXe#@hMc(9U>0@A!IgsQbKg6ZtvIP#)cGPWKvfv&%QWz4BU+wa|vM#iOG1iGQQOXp$ zwK^`P(_6n2Mw}-Ao7mZF!doVGz%C|2M1I86706FJBQzjHvm5acgTX{gz`QmkE0!Ac`0sEn5lkei``72VzRck>c_~x z=ucX0roBc((V~rt)l7Cn;S8LXn*kngDGpSc=G{lq5Bl!uJbYonly-phoP8s{TtE}j z?|RJZuuhXjy2M-LlbSNUUKv11`z9gS~brmp$88{T9V9ANWXJ*xFYybB3J|RJPtv_(-YaIFUQb zJNKX?osxN+lHl^cqIvaySM@_Rcbp(oF;E<-A!h*_Z&^yx0?8<5(&O-pf-WP2v>*J{ zRA^0(sk~S|BF(=FL;tr+ zfUjjFs+7Ip$z$S=>vLDICQW~YWGz)P_PVci(t__lmy^ghSpdmc(jKH?IW0Zb#dg;B zS2%B@wW(6Y#rM*dw6vGLr-P9&naGRsO}VbUKdP@ri!c={dgr+A{`ex0w>*1xXbScQ z7p?1vXlpQ8_L$>>4iD_yJ6z)ckNaGB)wlcHT_jyHuG@_ zc5ZgAAm3Oig%UA*jehrMCfPgYggchv3GNTKBwzWvxyEE7^?(Zakz(!W8eze79F9{N zUhs~26{N)O%6YQ+%nsClozI;@Yk)*?)*@E{kEY~C}(Ov!5 zfcWbd=aIa!Vq@w)*cP|c{*8&zNyeufR}y(sG8U|Mo$?-)P(*bYaVinG7ZN&abqB2|@9FR?esEWHS((|jt-7!YZTu8KZMcX;x7yfG>zc!DDI(WqGDO|Ue|-lx4R-C5&85n%6~0hV zj>wHqgC+eWlwL00#;%#U4Yy3U~6xm9(UPYi{G~g ztJUc!GuckXdaJmv`lcsg5~_s-nRTiVVUt#g=ZQHAHhqa`xw$BIj~k#6XVdSX>!Y1~ ztf_cWTT$*6ut~erY{6{i+WxwDld1Mp*I(HMZ{SvjUI=0jT)JODNYS!FMX}22FF*7o zMlqF%Nd|aZ-@B**`MTv3PQ4FU^JTNV=wh7=Z}Ov$vMD|+|y zz;P^!$;HN;UOz$e4o{Eg6AdQ+mG{{-#0Dcl{4&6siGzagVbtoq(*7K{DCq9_%ooo$ zAx#_==1+~!2Xr%vVe#6%NxD*AUvos}SIP+{^QGq_KN?Ck?`~iAb8p3(=g53nJC0yU z)RUpKR)3|!mtyp^GqM?Y%CGQkr=qVCMGgZ9t2iz7_xawg=C1B((F^QIxY1Ct6bl|7 z-`i%j74gA;x5Y^Z|B@ZRKKb*Ck6?c>nqjCylTI$qMVW;3cK|W!WqPOo~$ssVex@294AVTU$F7tlHKB z!wK?%ny}3RCUM@&$2oK6=J`xrm2}aAp$!-K%&hCm6LlpBzbCW*wj%^NmlM&rN_Au7 zmH(;@5)%2HA8(IF>ED!Z7Z$yt=8bhH+KCJQ1_EPX#D9#e=?^b0eSKAaK3huhg*Ww> zy)xb?gy;T2*&TaBb?mi17g?|zNq`w86v_)0%f5N)3SG_Eh2N4K1y8a!Pyj;Y#^;pO zMR+mP>Og(Y?wy<{S#RJd$*ifX^DYV_1$_fda2uCZVw=L+4)j#UDRu%Fo9`ty$GO6F`ka6Xm3TS30|6El;j=VditDsuK+t6e&P|x}{6#w9<=;+zW7xS(ttpojpR1g0vN2zh!-MK-VW^ZpF zAIw~~vb8JJ!5uavx_kWbtQHbR9uHG2d)loiaOd5!(hcdgAnEKKzkx67BD{N_g_w%Fw(!1E z9{i}W30LDk-Dp{b5AFv;sgA{;_-pKjg15UC#VI$Z6{P#bB49|R&wGCT{SypGOXcKzAGykr})#RRU^b`6i2_m4*P#;?>nQS%C>b;8|+3D zTa=)bXsZNEas~;fMS>&&8c1o3+oIcjtc&%15Uac15Y4T6WaOL{@s(_m05%bBWWNGW+r!adE&)LA9d!x@x%Z#e}o7ySlu zCxi)RW<@yqLr#7d+uxAexHaQSoqmHiKBKuDd`t2z$LZ*oh^g`@oPc1W7zRwvhU#j7 zs5w+%v2dXwGphyrh2Go}~85#gyCwE`=;&ILM0oCURF4x5ZV$)lTq z!zaF=Z$&Ev&!%FsepifI4@re?6Zh#+HzekxOjo}?u<-ADp~L$_#_3+(#cE^Y?Hepo7$27DSkVCy4uvxd#S`$*^Cb;Azy@`3q1c%mh zj&z~cM->biJ><7vxMRRDNn2Xn|EYoDOU9WYQseQ|-n24Wh2#Pm?~M^E(8BFRAAPmO zBZ=NypA0BcYN1>GR#i()GTSXr^4Z^;+3y*NdoNJ+Xrk|M7(2a^kRiMLcnNy6uO4>s zt%?DpLtRPy+#j-gcol$-TrzId+3Q~Y0FE4p*|sf!aB3lQoue*hCBL04bKXdoCD3x= zo#-=FL6Z#94qNJ7!Ayy<%zMjaS7C=~>j6J9E0T`t)$Jt}ySDbXoyqo;kQ6(9*mA{W zIBWqPCrbxp-{}^xdqrHTf>OiOXbZOQ&E_^hYbN)edhvwQbT`S@&dVk~u$@0Gc0==m z@&VyiTMN}241$1I9Ul#k_S~LSbDqVx$lpy&EuR*$Zy(UQbln3s*F(a1?v?lgfQTde zvOoW%CaPMPycm_)X)m*_Ws?|No46PEt2M+ZHk77ye=-|KieQ;ASs-+$_bEFXPIvk> zKkMU;ie0$!y26q?sF{xL8gF|XGhl4dJ0^$`3GJD2Ju-UTn6Bd9r1j&uN<>?pidpUR2?5mikiRmlXREQ2?Frjy8NS;uOp9jxvJ}UscE99QobMDm2Pest zAQ{H~>^htzfOwo$8_Ssw3c--~w+u|hOPm=3ujHv}Y}5RopY^$D7cFw~Asn&vXcXx| zjxrDrl9H>z22)!;x9Q$DvdoTmWW04ldB}}AZwOl+$)U-kE)KDiv7RjhNwbuiLut=l zb&Df4+eSuvI59*UG<*|NU>e%LksZKbm$fJ}kP~9;+(2y&A&9Iuyo zK0^hj+l249bSn6KhTvJ9Fe4u?QpX&Rq?2Z$*cFuM+RH32;Er{$VE0dIiLYuWAhUGo zO9IHYeC6EdGzF-xXuq>owA)DmfN1h556=z8eM1i$Y0GY}%VIvgsMyYA|GRyIWewU2 zCIpx{pr0=5w2pQ0MD-x}2%Hyi&&4G}_d~kOb~f!RKN>aYCtIwGaOGaT>a}`OZn4*5 zr?i4x9eP&1< z-5sw}ARhPgpX-#=GA+BhDaX&#T~7i zulaMzo7*C1YKVDhxuVWCI^9*B3!iI3uz;bOiH?vZ*56#Pl4CC`B296`&K);@j4@JtWIhdR{G~et#WpBR{;A{)D+?6Vd=c*7>c8)ZGC$$FS56 zDYGQr_MWf>fKcq1lT`R_lju8*k;Fqv`LW4cHVoMCc(8uY6LA)=lDN z2|dQt%1%F<@I{FNz%IxwLX1nsu#8#~jK&kQp7j@CuE<=vJD2-e?JM2ELjwUgqn)a< zeV$>2^%Nxu^Sc$q5014DDxoDmOwo6OKp1?ZVmHF%U#<{$nyO9Elm)-;KfO}&*RV-+>qHrxQ$KG8V^(&8}# znZ?(Fp}W+=RcU)g9i84f1%DYPdY^}@qW3y-f7-7XTX(Jji!T&rq1}cv?_sWJ9^gQl zHvA4yu22tnf~X)8nlTg${EY%=LD2B(ZvZi|lY7?FzZ6h$&Uo0P*{ebCgG=rA_kP|y zavVKlI-1;nYN2h#Pc(7jHtfH%pb^Va-gfS=RKR%>LmLRWI5+nf*;7>)Vw|emWm4|; zYLs#XDv%kTWfk4Z+_3ao-5D=TRU2_#>i{0pu)oWt>l2CfsIWc4&VFBC;k6i-D&U49 z!_`72wviQu{pVf*RQK}Ex%+1bf@pe%D15`VOAY;0B=@shfgm@M9EDE%mEv&@ni~|0 zTL3s4k4xAE%!{77DT9`b_AHoRz#7qY}4V|p`eZS8;>7n zt_Wg`oeoH6s+|uuKl#y%!WqhttYNROfLYEWnotXo3Ag|3_~^jmwqBxr0Byl((F~j> zdPrbp!o$0v;3T^$mGG8FOc>*iuN0Tz$uNT_`HV?hHJ^kKYCNP&EC z);I5vfk}Vvv#KzhmxcDb_8qIN-|70uYzmN3I0>1@LFJ>oI~Uli2*(@M+*9u_wz*!h z7L;cdEz3}(<3OM3vrkdzm#?6ttC0d4HroME#pO5B{x-}xcKS7w1N0ec^JGs4Ps-Hl z{enjh;QYEUsqpjiXA%;Qq3rA(1QNyY7j;Y}yn(g<)QoqxbQa@A44F3nmm%+h3_DfV zbQI~Jd{i$!oQLR8bYH&rTmFCRETo?TDq#}EyF%GAr4Zj+MRQB7!>o=Q1?r?l!ZH)0 z3mqM8JtttU=epdpkjMA`XD?$Hw6}WNpg!7kJ`bO&pYes=d0)exk90`#-u&J@^|ka4 z4;>w=KJq^Tph@!Y1N_%_Q~uus>HoTZ&VSE->=5g-rXeol+6H!3AZBjR)7|%)Wo}g9cuSE56cVNrgu%=iLJ6nl>rKv;I~s zyYm3sbMR&L^Ot#!zz~m;j>FIZtO09^c$fgQiZbOx$+^PqyO7W6a4_~~Y2YZ;o&U#D zgsG{xu*XWmu|eRl1_7Tr$?s1EgY1Yf4o9EFf7TqtQGPjvZ%W3F@Lj%)NTxeG2J>5s zAp$u-ir4mqU4{*uM34dtouc5+p}OYq8E_Rn_e2|{{Tposg?tRjGqe`=4t7p?;Jp6;u4a&~6&B{{jd zwmU%1Kqc^P*}cDtbk{3({iMwI@H?R_ek-o)50E}w2k`G99ibR zsMj;++_AD1M3j#H%5|qq-opeI8ymhnD>6CQZ8iIvJM13)ps@bW+dXoaj#0dWkj5GRWvHx-#JYzIKP40-wTvb6?(>ebJn9)|85 zfH{3s01!gl*m_l7?xORV> z(t#@x0sW|)tb{csDorn7`wLjSRxF3bzo@WeZmK9n7NiS+beY)0sdbY zaurBQ$(O*1Y1qy}V^Cz5^qta9go+#wjWa_%4YtD?n4AjZ?(5P=YIs0f<40AKr_BU- z=R*gh8Qx*Gebw6G)M!w9$)@{M!fXiF^6)8>9iYhCRBZXz$Z#Vc^2lVviB30#@e=i3 z+X)$%2B3JYJxK&Aq6nBD|CU3(b6ateg;@>XQp*jnJn~a*5>;;JhYasr&_t{^ zzao~%(uB$YhMaSzYtMxY@}!(Igsxf3e`3S_r`Cei*!;6TNIu4qZ2AHko$tIa0h zJb$Up%CFU;TnXiQw&t+$8UEj%8^={~hY2Lav|4ZLn6!3gn1RfVx?NjJ>FLeHn)$4U zf$3jsCft8{DUWmr{*6vr_q}Rt-Ny78bwMs2xY+)h zUi&99=%sm+JJy84aloE+*fT;ufS31UR^yY2CFmFbKo}$6a))V9$#YeF}Sp+c4dGRJ(&>d34oCLy!p z4c{ELUpiUD7WAs18^2M9R{=#T+yI`LTJPx%K#ij^T}nrO3yKne*gEGwG*2S6idpTH zlkS_NA%lq>N?xJz{8tjiXro~f_h`If_1;Y}9{FwDVa#R}^RW;UR2CqmNAF~y z?!#P(W3Q227%d#9NSZxB8#GkeXT(G~!!Ukux-s`UshO%`ahL4Cw4h5gZZ_PJT)^^Q zz`wTk+*XwDw@XRY*oSXSWjeS5Ol-Fns!QIQL(?}eYsI9lxaqLs*Os+YJZ%BQ`TVxg zl`o|;IDr3_SLNF4dyVncyA7rX!11B5car!4gjG1HM8P{Y_e-UrN;{n(C!c3$fwPa97Wg=R!9V!{K1rky#`;Qz@gwsk8P!pp|n2FybNAGD-*C%I1e zS6i2Kzb`MH8*qM_-yGZX$lbW|9SBA4^Nca(F$b2LQFZAR=^)wZLm*x{sjPW;@)PZ5 z|H{bFois#i`xIu>>9H?sa{V0jf;p0SF1APr%IWMGA>*-Ut0a}=4#9xlk$JRU?F6E`&5vn0j3i% z4N{J_i5BFT1v4`$R1=%_qYmm*)!l8EfTuHaF+tJfG%AGwoMU2e-KgfzsoOASH(g@4 zdZf-MhSZ`Hgi?m9#qBr=w;*HiU=tV`PPHORyA=HZ_QAvXSqG}M*l@6za_iLtjXa{KM$uQAJ=z!OG$ zssmO_e_qd5SBY4()z;3;I}b`>TsbFIedAD3dfhdG5*BpA8EDB+OyvXmG?V`}ZgoT` zBe3`TI?*UmS8(^}gPl-)%6n|>JlJoEF;WVyREd?xAM zZ3id-vb+Yd&-NDBo~~BKz=r|f8?rp>z(DDN%FyTWRSu2Y_=|~E^m8^geUn}u($F{p zvzq0)4A(0j%VW<<%~Hn1U;F)^vR!Z#$XD08$co(#Fc-9qmO_n-y-8dDI{2$;$;L83 zEy(@h0ySlwh=K?kB@eYwGRu5N^9uUzu-CfZm(?Pcg?Goz;?NjS8)i09p7ylguP+ESyl2ea~L0Vuii#DQnQwb0ycIb z>_0|7uloQxUQI+7)g?i`Z!*ifv-gmE?2obbp{HidG{3wuxJi#qy4O9E6S%`yv|^6# z5W=(!Kc5Ia=&lK|XAyU#+00Jfy&R+<6BAVk^Xbv-$LdJjHV_XH&)0W?OExnEZMn@3 z&Xz)zN`d@=_}Q3(Orx;G8)BE$dhjr)9UDB_D!!(mZY$90COI)<#yk-1(hN`(vjzFL za;FN;R|}p2UCG)TZ2&bzC_x=F1IIR1p#CZeTB_Xg2fB~W&ip1E%8^4?OWr~;$8UHX z_L8w{B~aYwnZZ3s~trK1hkbq4CKS zt-iu>#Y>1jfnmsKj2t`s`dug{jiH+fQfWJMRj}_pa*+0(9#^lM1Vyq}o~==3?t22# zoNX?I=<-6m%Ju1Yk!8$WY+KRacO+rfvN;#*^yHO2f!43W2WO%|H8CtmWU>tl)qC$H z*DBXTPd>%c)fPbw+FnwgaED9*7m?lJ#Q{KY9>KV00%qS5@u90j0Uq!DI@WGL0_Cph9FGs z7*o+x&__GR8A{YxcASo21+%R^HL8hYfh#p2iyL5O%9z`!I8xCco~yJDWD-%Z6n`z%6sK~Ff3%%Vhxbxzx3J7|CAzL(vrTZII%a^cPV%+ z#d!rgX<{s47bwaM(<%YWGODe=$&ely9SpLbLBV@pG>~sl|6iy4y&Y@tvELvZqb^^3 ztoB*6q>z#uutkp!!T`v{cOeY^?|<~~rvv{5&*}cBw*D@N{VSgo{_isb+P>$aBKYhh zi=Mqa=F%o!l|SVW>AL_{fWQ8nfxIY&|LNeaEeOx2i-gKHygcF*{@w>W_2SPbv;Y04 g|1GyE&1&p`qSp@*aZ+0wi1pR3Ybm|^)jaTj0GWolaR2}S literal 0 HcmV?d00001 diff --git a/src/_assets/img/performance/tracing-diagram.png b/src/_assets/img/performance/tracing-diagram.png deleted file mode 100644 index 632b9dc3576d2b509f6c16135b13e068d27d8c96..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 23185 zcmce-cU)6T_b+TY>aifl0;q^6R(erDK){ZQh*AXtNC`+uKsFr$M-fpFP&$Mn3L-5? z4VxAP=?NVIBmrrg5(%9Eq1+w7=Q+Rkz2|-Zx|h!;+1WF*X3d&azO!cbn}-inIQAUd zvt`Q`j=OhmYHZoE3)r${Tf(2);U_l;2mRo$qmH*8J8IflIJ%fPAh%pIw=+eayo)li zL~0;S%-x>VBCl-Ovi*XU)?>%VYWL;M>`-DR>^fquD0{f|mMvG5UF}WGY>E5HTxULr&f2|9gv#t546nOZOr7%1(cOeUU8L&380XUCMR7{ zwrD`!RZ(D_t~`9tek?9z|6tQ-qFeqeUeSl#MI8o zQBeTqbYlo8`~Q$d1Dj%k1t#ulVlOT!Ccz%k?}}<_|KC+nsQ=Uk95s;t<@bL}7|?RF zM~Z780XruLGkD@GPP4nRm%rhFG;y?Z(6Y0${Vm0ZPwgD-fTwo$CvV(fOHJVC|;Cd@HHTk<}z|jP4hP-=IQ2?eRW@TkAFDY^9#-*FL<)meA-;|WRb?w^4i`V6( zB%~y-$z7AXeM$Ov+naV~PADYW@poJE|F*sMuWi{1fwG5tzKL|Oaz>ipcCbU8Tpzf+ z)xYy1`>*si+nWD7FPHzdtvJk#ID2~kG2Oo(!8*jg{g3j(KmMcsNHnbT4zO~+d0N*7 z+su}S_qA?ySYeW4a>48h8sbrVGv#NH&(9QrndX;$Jy2fu_sRaBGe2&V>e{BG?dc1?45=RaDRoq{Sb^itd%dDF1qtODG(n(s{m4WF$|eVX-9%fmgL9X-{A zhH(3f%#R-NVPAjcfeqE|WyQ6O`sY2g!9M@cg_4hKzl@gW6W(8#fLc=TSS<}^&d)CdxyQBCch2R;G&gioUc7H3br0fGhd$zFs!V3f zKhN}LR}|GVd%nz!bxgO_&G+SoU(#NhqOjV3%>5#Nh{@`~sgG89EiEq5J2HmR8l%J? zeWCF+l`Zp~4IU0b^W=BVx(a;+kHuR2$~+YF=F4zuE@OOpxxaI+yN8+nCONTiVIc2~ zQ}mDWrlk?W6CKy-ui*np?-qtei!HsMyGOVI@1`1jCc3kRySrwZ-Tb_xmmu(0$-{5> z+V2F?T-}e4@%bYid93M)uFjte#FFmz-jN;>y%v~genoGttG{LNtIm6MiNR_j%#g|k zbEC&gD`&>W;+@~ue8_%>Nth*5e-sm^CTC`8?N`JeLxpLJBW)qxi8IguE~TicrekK{ zo4=lSZ*tsRQ~9gs3B6vI7V^U<@d;V!rBe+~E0grPispTRdhR=4&&f7w&t z*4@@S-}=hU^sS7j-W2}$tdDxlWepasx$h@bS<$>WnE#7Dxwf`8nR6v&%NEU*S<=ThR=+MN-1uYr%ll$^=2ohDQ(rp$pXg*{)aZS9 z^g**#0HseA$tY!XqQ`0XVhr2n?+K(l1@^_s7mGgj7Q z?as{%m%7bZlo?Y+iDyF1S25Eb-rIKz2nYz>+{)Mne-qCN2$Un>oArLieOg?ZlHt13*P0 z^0fqBglGs9Z3RkE91y$l8&GwLu-gOf7VewfFZ}V6{e=w&IetIbC{_J`R)Sj?Xown> z6z&1uz?5T&aj07DA`4CqKJ6O5W5@Ip##40P^!_x3C0dT~j`47;X?vtQdXNx>W;7qK zIdM$>%|?HEF~v&w2`5RIArqW9DGsA`zhB4J`8knf#r1XB=DYhk0pl29*)X10JEU|G zd7f=p-+GuGn85~Y@ndRG+C5rK+Pz~w?c)(iqEXygvOyIsyD}0Ck;}u6Fbd}f2)+hN zf}Tf*KF?()a@X@~hELmk*_}{o6zgPZCUyXp!wazgLeWA}j)hA>Q4IWV>r#-j%6uAL zrX5;n(rVkPtj&GaJbCE5_AB8^@#HQ&r#8n!GR75!Pu&Xs5N9)vPK`=OtDO~c(=h5% zFfdNA?(D3a+#1k(Z^`vj&WnV3=vYBt&e}13dU9o5`5J#3Td|0uM}g#9#oBd$>pN)} zy(xNpLWM^>K2yL$GVM57&;%q_*&Fk zb7~(&OD)3TXDg@G&Upvoi9Xpy zZ;!4kM;Ou2w)0zU2cq7&J|bAXhgX&p9y%$YvgC2$Tac}M^fYi<>_)Doczj$7;@Pl1 z>XRA0TEUuexAm0J<|KuH{m0rWBT!#{T|tQ83i;At;bJi8aEbWOmt0B<^c;=>JXp@X zr6)GKY=>$;jgqwA=X;eD3}qM1yfYc@u3U;}JfHa|1rs=D0qiXe5M!HxDUJf+$U5EK z(F&9+o!yv{JW8|sfHXb*Eriou#S4|vwW~rGNMg(7NLUw%l?Z&KXh6CpIfeSlI=gRy z>wJ2MlF_!g!RAlr^D*f3JdO-*-e3hMEhYV`X1qp>jcGyJq-Ap@0=5I9lQ*xB)woqR z#&>o+MwCB`UsaD1;8M_zs8GA(r~AMJ=Z60@x1;wK5AfzsK$4s~yJGjY`eFogMGkb$ zwv9Dud^~r5dqDVH2AzlES>hGqGZzyX5bkqPchv8ze%2QRvWwN>Dzkkinq!amvdSjA znzEN4w3-h1o$>bEiGNb|=HtF~>XEQZfIES^V(ZZEjW1ZC{^c3FI$SRk5*sB<&a7SS znGLjCUN&2M&7Qy^Gsfox?*ZP)VGgzzi!!i)yngWxK2ly=%tDc9T<188!ti$mh)!f` zE62*js+?PUwUU_`8a3|G7_tlVa=ok_VbX+JAXe9W@>+?etf7Bc!{vWM?q-ZXCqiMD z_z@H+N-lS%Zya>);3EjZ)>MZM>gR8+8hE+1d$#v8ZY=2Ywe-H& zAtjrl9^PQZ%SGr~{RzLdShPEJfQBNj zk8bb-s3w&Yr#QCbyntAujfCv+z}t~j)Xl^HjW9=!hk6s@V z9zhJ}7Evh*tHRY>|9DkjvJ3byM{{C8<(i4#+s|)QZuQNOy zO)sDEJ}};OREJ=C$M6w_ple8=rXbpm=g7Kv_Ysln!WZQP&KjRiAqA@iykz`c4As4j z=kPa`sTcOR(Lsv((JY*)uPuKA7SJCd|fbMw=@8^Y{ZcJTWFO}P^9py6p zsa4E~fdl!b4}+q|!ez8WM6_`sL1w2+(nLffBG#(PTRMebAJG`5MO}dyx;#f*&P*f- zdsXgtvd}KuQA>?-zh>-@@tsdltlwlkDxDEdv6>mV^oGpm)J`ppSa}nct)t&AY$zqt z_@O_AHBi0c_fb~ z2z4Rc)(}&6D;bS!%Q(A>INrA3yBW=zbW|YN%lU%MUgH7XG3j=A5%prp3W;L{6kc)% z^=TC(FZ|Y}zV7OPD(aUsYQZdIZDcLm65Aw`-JOmerY4GU?T;=n zcyNJy$2dv+BJgR$dcs^(xZme3@c&(|B;uqJtJ#+0w#q>v$6(Ll1?v+2z%YU?-!}!W4al|3H!s&A}f1!9o z{dzvkbD34-)J>nt_)#Ce^7m3ZDLs$x+f4q2*9$UeRQE2Qe?xDOa~xN< z;@_{Hb6EDse!scxS}(&m!boKGdF}GM3EF2QhYy;FK@rx3><}A;k$C%oAJ;cLrh(3_ z5MsWZP*ua7t<}^V*~krla)~umsMetti50-b=EX;Cc7tF3Z-*gJKXzl?UG7MZ-j1JF;$i%{xA`>=S;#ahKf=gaMp2$ zgKoG#SpI?XQ1|W*T(%n-8Lsa zBq}F^#HE|Lc4WRpI&uXsL1set%5Fcvirc<_5Xra!7)KBd^T+0mBmKc`MfsZPaEp77 zNro-KaOjT0MuKYn-1YCz(JMt)k2`7u#;e5YdsT?U)Zx_0IWrh~Lh>lwCCK@_u~^*o z{cIoxU%LUmi@H>!w2SS_tkbZ)65ENWZ?JPDlIp4GDUPygMhvn)Q4&TSlH?~0n*^>3 zC3Yc+8rbgzrb<`A2m{eht&T|DbtlkxpEsKiAFy6SM4$U$uR$8SnQmH-XI?=-w6kQS zgscrv$$Y=mrVCq>DJ~CF(VbNBDPmg@01=~Kz)L|G(L@>Chh$P!!%siIdR6h%YhZ6w zN?tTT{$6miKlVCq@>6ERbZ&5drjo2p2Y3UJh?F2d1H9K;3W4vr-BjjJ3|&D4B`Q=; zcLU^Hpj@|!zyL+s4U)V3GeL_%1)auRCJpSd)&?d>oTQFkC^CvxxV1nD(8DkS zs_bUghuDIc$~4OymsaRP9|h`*)*fY@<2za9@U>v6(1-4r5_<@Hj{57;sGa5>b94vo z{OX8$KB>#)&T$v)dN^(%@Q~tH2=j%D2eMCuTxk~dx=QVv%Xo7=w(tTc zkYeRC6?RLnv4%Z#?9+q24<3ZojlmP3;_e#lbb81&c-$^I2wh?;;qt)1fI_LB&(9w} zw3cs|FTuj~xZEdL-^1G1UhIw%#Qa^R-5PeI|EuRy|FW)Dx$z}I6iOhrKSIAhvaNs| za~ku2A~|GmK%6jft1kre*$uHkyjAD->rjdVdqx`&qlI@aCFqNS@h4 zRexjE0Y;Q?&_||d2;M$sI&6CGv)ZGq?i7*X2pACkj3xG+n&Z8aH1Y*KpSKjXd@F2l zuBJuzit_`|wpqz(W%|_6N4@IViHD9lw1&mU0?cPe0ZF_kDU%;+LQL@Z2TkaZ+5eaZzKD z8W}vSN=R}2Y9|hB6Ksy^dUh+KMm-O6_rQJZQ`P@^MX9WLKL5}&$MZCEz%ow3<<1$$ zX3Q&_XvKi>3iNkdL=m=kDAukYyK%X9mV=s^vzX&dKE;~eSW05 zulI#H&iVT(ct)HW5N?|9rkpF{|2pq^4_8Y~SnHZt6eNnqbfbuBrKaZ?F(6BXbx6=G zGcQfI$UOgwkAFgeyqn-eHqTJHaof4}8*DJeIE>moefa3C+?a!Pg*I%T^ta${w@3P( znajnVngI?X%!!5j2oqxI3@d#fw1;Q$JlsS7Ky+q<69s{J&%pR4<>wcHKwfadr(gM@ zQFHxef=h{KbQ4-TnDLs`KMz+YYEhP6@iN#EX z3F_*Kw)hHMhX9t5Ts>DC?R(2y6XJ>64S02P?P zrkY4pi@medt~9yJWhrZ%k`+S2n6!^4EK1i$2u7|32ZGNyNnDW};`%h_-moyJ3}f4C z-G!X`_Hf#{!LJ)S?of?DjWfo8VI8w|!#@{89Nrn|ZCT83klmlAtzr%sm#ti+VCGoc zAnsgnKuu8A*PeoydKw^p;^K&<8Ls4tp4{y=k>?|FuhMnXu{W#~n4P#r!_09Sn!t-enYwF&Dv&zX}-#CW_f*c$05OoZKGLsW4~B&x?V{0{Cd zBW;2;#KORjpcb>DR^8c!Lt$t&N)-Td)BiWpXS z8b9*R^W_wY^9wxo@}?<8)c9LRc(KM*g%Z=BhD}{)`C|kTQ655 z%HzJW{eZ~aOXZN)b-TvM;u_A2issT}d(S(glXZe5jE8^Uh4+4)=x8{WI(n%xbF5T$RQMtWAq^Y03cCwu|J7k@J>Z;Ee+fYrus5o3+m4S5^j&EXe zPjL?Sr(esA%{$hqyx)2aHflKNoV6rrH}M6FQyv?8f7DcbLA1Q~7aWG+SX^rYp!NsX zy@1m`>lI~*YO{~kr(7hrnQjmSP0$|;yrNC2Zm}aYn--bKr`Oula)NOA1YyBcXYhAL z8}wSR3Fih`n06#P5X09$zt_85M*!@S^}_XH_QysUjH58i0bl*tf!Ux6k};AWi&W-Z z?4=SGZcO76mM5pq017hic5%Rhi%0?6+au%W)FcVm;=w5Vk* zno8me=KEx8eS~28^)1w~F&(1lS8!18F4WNx^aLjhb+E}Kg4@6{DSS~k6Ds3$Dm_iM z#$9F9@GcAog6w=W(h_$EP0NLAbd#ZYuVvcA=?QnHi#jiftJbjfwRe1!LWpz`Tya$v zPDPWw_@gP^V>WoaN+@?MpZh;b6Ex&AEgJo7JJRK}#5~d3&67(+jnF^SSUhQ6Dd5(-~zD2(dEHuahrQl1^V11%)vFb{PBN74U_Ldr&Sr*N0dK2;tOg@ z;5P7}+2h86Z$78g)N`K|fn;Fo*?79HlA9IYfxU`T#@>@Y{8vwI7Kfxm&f>4X1RrE< z)G-wUEzp>6vzIZ{#xZo~SChS)6;094VkmjeRQ=TvbgCqr0FqX1^LdJToMT@5L?*%g zHjCTtG7MAJm3c4?R2$P6x^wcc*Xf_oJp<4|fER(JhfQ+LW_buv)U_$4JJ!_&)>lP< z%i*-BT8pxW{6NqLdN}BDQy!3Zsr{_DK`TlX9W5Nnt8C)>L_Y4SOQT@+{_olQN9_z? z$n+pVUsF@_6edemJ_rV<>mvZoad~30E#~flZPjslDZ-hSh8mh%3Z^gVu2ML#duC~y z<3#JlP(3a*{FBIbSIteaxaELU*4u(f#XBeJ;e1)g?3yexAegj$UxiV6UB>j;`CDgzjF zhx;X0GSq0%*T}ZG@)4}TO-VW%1Z83+-+ZD!TiHIQ*G)RO1NP76aIX@@Jau<_vX(KxUP@5ziFn3f{Ym99%mZ&qPHxcF0s_!qT%+3gFrs$+iUP;Com&T zcGrsB0xkWZ1~1f56X2YLK88B^#gS4L;8gSkU2(L2^=92;;momcv7=#@52spphBC$E z%M8lg@iM78ug!AfK!d!qH6G?ovR~L~q?mFlM-*KdFv!@YvDfn$fH7f|s8-@U3cjoz zNmjl4h=dr5l+pG4wa>Ohlox#XWVWT* z|Kl2=xz8<16+~y-E|Q9VLOmx>99%5?Pul5wDE$w2*(10UZ8A>XIVKe+3;^cS6dw6_ z5cFtM%aI`4LHbPYX;vW%nQW1b=9n1MUMJi$m@*jC{{EDl+ zQS*7v&|UmR%T?CIS?4(ylk4$1OMfGXqU9k4 zGFEunUzZqFl9Dn{KaG6evv`$z9`fgjmCx<_sLM`k>1pN~C_HHY;E538s}=3usgnkJ zB=KgYjyx>8tq;rz_56Uh7Ac0-P!K+{2;Lnqi)1Bb8sK%}s}XgHgJfrQe;N>{JpaiA z-gUA3P;?kNLFS}hU}zSrC>G@unc=)vb|52r_oRc<4>S0UMMNVAd%g+Xkn;D(q|cUDHhKu~Bpb zAi4e22f8rOnBYC}(z_htVD5x_n6grr&0jb|jOXAzC%5n;f`eCPWakN8nTImQFy!fa zH>+291d=yH@BYZz!*aLBa?n8%-lRTfjBt zWEhZ)ezIegw4XIaf2?+o-PbW>it}byR#%>MPtvNQZGpIx!T#?bnI3j%8wkcJj9CKM zdhZAI=0~ri{if*JC^{Lp)Ym;|%vE59eiXxM!9f-ASXL{Ml$pu5h}s>ckf}vfYYV#4 zee|^_;x!lBF~`T{uG)2DKzmSxgqjV7@y=<&%4q_YRbi!cQ#Kj2Ikptp&krwXQ=k)z zzFai2B>n;^(*`#i4xT=3Qx*VLS};Q4?Vl)mDp;Px%j^>?8nR;SGmb8XW(QVfnfKPA zp$f^cu*1(|uL+KmXgIyVBZ;peV&(*dJT2L?cZ|4_xI==CAj4VpEmAIb*L-g$YHEG` z!D994idP_r(ow~Kv7LK;1yya^)^M$FHvziX1}-T=p|?Za2_UDe#&Zu>a{5N7XJA=q zqNo_4E;~En9<_&xD;iVVC?SGK3!KD=Oo_hA4o3UfnA+)0aA2SVJAioH=NB$zpS+W} zHv?{c4dN(mzmTUUEBD+cE}a9*WOT2)2)RAH(E%e|_5h@YmniVkqPxDv=h(q0ItV7X zH+rmor?2(+uw{%i$NjZx0>gx2w;7Kfp%-lU#YVNNG%5+J3%Acv+MrFw0J1!RpbA_e znhlTfVQe5d#aU^>IKUQ9C8C@fBI7dTKMp?;`u(Iu#^qLa6uf9$s-^P!iF3Kn%X6;6 zVMM8Bi4Dx>;y__tG$B-u2LLXSb8qKLGpESQXhu7{IyTq_;o$rlsRPEG+Byw0Ayj`` zlHoLiuz*SQo<{b&c##{+))6;778K8=NMSU;bTgtL5V{{mJOLMSS1Cet0Wny>}Uoo}5 z08P_(Q!;}gt~aaHV1-8e&50H6Z}eQJPH;UHae$Um=7Mj;4&88AUs=GCn32UFDPxHc zxteH;i^gTWy`o}!8HUJ#5L|>O+u6W5Z1=qC~LJ;Sk!-%*sVFnIf>qCJ*HZJ1k zH@qleH-#UvjbZcFSJDOjNbAnT1FrAEaM}7(tS^|3X?xKI+i#iPVXhXc;;qSnbsd5~ zENmHhR$lU>6P1E&n-$y-r0-Q>vP1rte1jW+HM8XK#Y&QSAf*><0fb^bkhM|8Ndzh}(`zm?`6**HH)7rltqVfs6zOW;`A>7^n1i?77x2YI_%OcWjjAE{wJMnnqF(at z*%!8vM#_k-j`IgsSa@IGC8BC9mi1$A1?4S|aKgJIJ;Fv@(x|Pp zQ{KxxD=SK}%s7(p1ekPW>BsdABZUzBF%(-RncKN0kanc(-dl5KPXdq5i+5A(j)icr zPM}&n$2&h`T)#99LuI*(@x5F(5-zU`6IL2}1+u@G8agg zZ_9ja@5Hvinke&eHUvZH$PCYq635*x+u(+zFw*v55)!wzx1h6fBInQk5*gC6&abT z9Aq^_HO@2GN#o~H0JHSU`jBibsE;*Jg3ig8pL6nIsX?zu1_o#2wkE3gf=`JhiXAd2 zE61iq(G&^YIDoAGXFxAv&;}TYuyZ}lo^h$+i1%Rm$Wdw3;9-XBU)`4Rex%c5onLeM zI~a$S^3>mf`{$+2a03A2eZpIVW=wKaEvE7w!&P>(aX2`AN-TqNDmeOm{%fJ9ylu)A zj6;q9HXj4v=K@V|xd5a5%#W&p@d=@p?ofhW{W+~omUlw;-lnDCB7=8CS5aSmK~5yh zLNOgj{XX(V(SJJEs~GV!o#Hm@MnL`#Ud__1HL=CLC(#mzfJ3Azr>evixMnO%)f-D$ zk1DV~wZ`>ZNMwR_ZO49%^V0A1ysGO!Jd4YIPBS4o*)|xup809%5P5cUzToKU%#sX@ zTcTRq;65PzE)7*xY1rap%Mqj7m%?sH?vf;faxQ(Gs_Tx#iAp6D(s8{6I&-Bt$8k8r z+Z`g(6;Ti|WmrsfFjj%wn(%n_S)4S!(NhTCHV&1&5wUvkm+bg~uxN0p_CS>cIUA<- zen)by46HBY9e{VQ_)4p?ng!#&U2Ql?lp4!Qid3GuFG~#PIkbx8pW|2Uu#5X{SdmcN>aoSWUXiDXB<}{p!b!hp z-XXjb-{E6P6zYVM7u|4GFc3`Ph4w3rqr9aqyNs&&USxXC1^V=?72;y+)g|)mTx|(O zMbd31SCHQeES{4}mmb{2?389nb?MeD?5@ihu^>9xvq>wgp1~_The1&eywm5wY+Tl~ z27Ir%jD`Y=jM`*X^-|c``CemwK3}@_EKVuwp~T`_Ox=&1kR6a~WJ zMCNiy)Q?yo4f;SZ0evqknLWV?ujqP|q! zZ*?R#4~ShHz6X^7DUC8o{oUCKDuBjsJ6hy$8K&kg_}4qSFG?ZZD8+%sMEk5`4WfHa*71 zW$AB$6At{Y!+dbQcv}=JEQ^TowuS?LzzeGEo~I0?6Fl;wC%8g=7u$?1yS2x_^r_S& z;PAe^?$=2q%SYy(87z0tq%O1=#p}C7G!y%d?>( zcktU;ytf`*BZ-x2FWq3(&BLu*o35{Tsu+uGKAloA0iyhEODB99cE_!3LkqwffY)&y9 zT!=LPbzngc0$2J|cbD_Aa;RxY6(14UqER;cXVMN6I&$5%dODwoM@9`=6@p!vBzfk1ViZ z&icj1!2kI1Z*_Dpy4W=Z!h-nSi0p2Z=wB9m*-*E4p#~YZ5fKb}^rzvb7a>w0SddVfm2Pz}{NnA;?iGznG`i$M?yXP>KC+PUIHueIDyMx+2Eyet% z>_I80V8o=q`G?`ubsmYUCT?xcGg6fB z^9zXGi-tEM-V!r$W%C);b4Av;`{?DDecbY;w_JL|8={3nJeZ{T7y|-5f;-6VxCcq6+N-PC%9x5-} zkNB|{gFGZ!`$#D$4q7twpVj8jY%(at*YIsb!xm#t_(+M4vhF^pNOpwnYjB(-jFs*H zSMrr%|9}YD?y?$zcvR6_aEAnbz;livAi+GmVL0R`Xd3R`1Su0}L`=07qm0etq6}If z9fG^+qje*JwP`1SRqdvFb1BnSZ|O{Q?V)L0#3x=RWx~sUSlLCD0AAHDYJ^t$e|=3f zz$YvF_qtNA`xJT+*ndE>f7ABZzSOttRa))*t&R;Oy;11{*e`iPy|b()BAX`lAD1cr z6kR+yI_{j6{rPhB54KHGVfJILJZEizeIR!rNI0NOIA6&C2adB2uvt({SO>x8@KH@M zN=fVNk5UDnb@$8mt1!fkp;*aP6!p~vp9tLKkO(uBY=v|Dfuu+K#{A&;QUb5(HeFUU zpbDRtN}3z62~^lOcW1wcQ=R*UO)caE&dr^d-(DB!#SQQ& zWHrt1Nu(cNIW2>!W_S`YuJ+KJSiXn5MekZGEWx8hkbh~wbE(NkEQ(dbM?U;Tf#GVt z1LDpJuy_k9496=lu?vOy3*IdH+Q?n+mA(YeMk`>=zCjrHTq1>R5Z0u!)}G%bIh8Gw zoH%HsWaR7$>{__LQSIdI7zw0m1inG9m+JbVzEu)$SLH#IVNw%riwqWcoWUtTJ#!{x z!uUj9iZq$aF=g12kzNcnIA)GUW7d3N=0dToJ4TRK&7z#k(7eKM?=yIvc=%^<(`lpy z0|%{O40+s`DOq+^>V~qq@`QJZub(8SU^)|^)#^)7Ji1d7v-D-}+IS(787hbMC@W&r zrWo3&oKyZ5WWd%3*qCLNXRGDXg+0go%D}N5DkNHoY32N4F!j-g6K>|Kbq-SO+;c7_ z-g)hLLIF37lvls%djA1<4;6ymBTHF(tcYAYX$H=PEuzH<^EO@n&`PyGyvfc*(=x?enG7neuSKc5)7P-+zBb9PH!_Cr$o}qR~N&TNJ0#h z0peGGanjZdPbI^d-eO<$v&&O530Uyb-ki7W#FZUlOurF1zoI;P0^S}KfKPNctZ6BY zc&*S09(}KKp*=nfMNXxKpUd0Mct-CMce6|?fENT!rH-%giXNZB5lm8PcZJtT21q#E zRNB2_%%uURVLVpm8o{PHawCxjCsAY{fE}&y6sW>ld+{n$lfMq5jOFxP+0Gm>gA?8U zBe;^`@2Y#qh;Ryj{T08}Q|GY>zY z_O38&oCo1e{ox$|rz<5le0VRKF>|88J>ojdN!`dQ$RgZGlF_|NE=7e;OE7a%oy213 z%lKdrX~US9me3_WgL~s!s+FSKym63XF1uxsO53&r6LVkNbK@PF;L$SIU>br+lCtlQ zIA(ehOEVf2Cn1ZOadD@ihmXCTw!(QA8JEOXNPa%l1T7guuZ(5A^^>!O)8?SE%bM&C zB8f%?8OMo>Qm|DwwOt3p9An%G?P!zfPCsTF_?<2Tsxm2eqF!`;HY zt^);SECmK$n@K6QfBfIy{+GZ%X#IQje-Qa^OF67U>Gcu$>ebh3y;GMqIA4zynP9`K z|8{I+i5nPskWgm!Za{Q>vwFSGT*-BX&+7b63j{tqas217WyCfI?-5b)*n>Sjaq-*_{W=rjz!RkjYbHstv)w;PzPE83eSeqHw_ z-@?UT!oY-wLL;Ms`cak%^5K)#07DCHHVw8il#o`K;+5gd9txWGGS<#`{yoU-QP>5j z_E4(`4r87kY-jnZH!b_65n9e7`)MeB49~5(PXh8kg7|3x_cnWo4 z_x9&;Zl72~k$X^#g?3n`scf}uQJ1%12yUu>EGfKW=yuF~=Kaz^`!8QeE1Ch|^Vp_R z=HhVHsst2K0;(oBwy@QfEuZ706_+>E)`;TI2G`+Nrd&WYLym4?JT*@S(n=NWpT+Os z&d}EXsg+f1oCLpsx2e`WScMy&1A)vlG+EIB2J|=h{+R>Y1lQ**5=Q@?cvQBBAWtc{ z_sKza0EVZTtO6-b;rm07#R@6KBcv6cD91eR-)&rm5zn@c-==t+*Z2kH*mrdmNy=UX z>|iXr!K9j3t~Oo79Z-1I^1XledN=R@;8zPS_7`WaTrB8mcUf@WMMO3^H+pm1?^2qK zESl*hoKLKDajh2`yh2==r@M|Wlx-*yn=^mVI7cu1yuuNeXV)z70Hidde_Uz?bH&z+ zUcNaqh-wm7@D1r6%}yF{x`XeWv-1 zta`r7^mMkf*>u(Tk$;|+q8%4>EU<;BWTl0xCoq#fS%!fdE*NGKd#b3GSFqQ4`d3#@ zG+^xU(u6h0i zV9e1mQc}#_uqwhTMWRr*?^+#K-S1CNk<&fyXv}(|R62hQTKZ0{ZOW88{Y9b6TY7^g z{cm`2NnD_a-%|3zHht?_o@tB~j$sXur|AJv2EU-5S!&Q|t4iyt?##`d-qzM=aD7(= zUZhL7RIOkVbN3H@lGDA)tTy%>50LW^o>gh|UWE^PS^&K2q!MKH8AILB<^fW2fcqG@_(e|F^`cCK7m4f za8zIiQV*a)(6m8R6rekEd62@bi;rK#Pn^c^$W0G#92tX=+Gp#W#uXNIsbkoo;pbqv zF(O>mn3dO}-P=+;mL`6=h1~NjR~^f`^|8ddbMTpJ&XThVE*8m{If`GR>~{nV;xQiG z0?8s}bK91m9F?huBvYL;gX*QS2;Wm4%8S|-vNfqoZLkSvVDaYem8c=P;={mt_D%Gr3ZWN zgvUQ#p#S%m(c6)n;#8jWg5~=9m4f`Ufc7AuP(lqB(Fg_w*&JnSfkB}iI3-vXD7ckefz`i z=;(=nH*Y7_o}i60Y)|;S=t5Yu6GJpI3C# zWY6?!9@>7x`cqe_UErRCgAHS%b}cUV`>Jlh<{{`^&nqPpQ-Q5|IB$F@ran?=<$mxj z3qsbg^IBxP*~2Ce>H1e9gq+*b@}`-{rERZ|#wQ3WUr1Aa&;k@Xovdd{tQe9*GSSd-;kM6uYYQ~`@k08ns#c_{Utk?_u}oUh z+^_>q2GEyUuN)2-Dir$zi1QNZcw@(FNZ)6t6h;5!AuSGYsuB+0RQc2*drY{I=UdZN zn<^+Ko~S1p)HrZ}(cUAv3x<7wXN(H4<_Lqn?7t=wKcr9aA(80sjP?>DhgCXv@vFLN zE&=h7KZh;QTYK=f=s(~+Svw$ERP9__;L+R3y<=^nJCf-819<;*z8ir2_-G0xkcJ*_ zI>xbAcTT7xC-1;bq*kGMoQzU1E)>lHoc9pX_sb)rB3)Bn6{o2&O)l%B+}-fjoG)fy z`xd+Ii!ISLNG`UYE$@4pr&?-YP(0{;L7nAP7w7hvGOx0N9nv-{Q49t9H5#5Dq@nvJLwDD%~1yKkpCKkLzrNJuL{fO-fr0KB#Cj- z)H2RIRVw_2SL$ZIrBS5AqsY|<0%A&g%@}8bZza2?s7`R6a=Mi?-H@hu#*af@Ygc%_ zM!SiV)6YMGpPMn$inKEpdQNEKBIC#VmD^alUBB+~JA@JRjHO>u=+e;?Wao?d$#<*a zA{C*F$j@tz-Mu1>Ti2q6MGZTM(!GQwg9lor>K|on9_Op?sb{)oxuYg_)UzU-@oLi+ zkqroJ%GJoJ0(dgdOX_+scFvu*B=4Yef6xJvZHB^+KbniS^g4BOrW=u-;rKYEgre*3 zVkvR%;DLiGA9U((2oUI6X_xP;?XP?>#>*4@{17v(O#9d0Vu?!vmu)+LrOJncRl0$0 z!UjIzqYBkWv2HgCAbleX8Av~VyOPJ3>pPFM_bTmU+F3x(dM6IQr#^ptfANwBaNaFf z)eJ||yy{V<1?#qjDQ53;yc))t<=Dz^hh8zwMo-CkCOX7Tzq>Ob-_n`) zfG@n`=xSv3w>KWrX;;Sc)v$xI?N3BHgnJVo?HGA_58%i;hNY;=Fkegu*yex{k$O2` z$ME01UW$9_m?5UPxz+KO2vPN?1m4m52(n=PaFTjj7|-sTLr#18zZvj_L;z7e_}hVv z&o9|Ir;_q9d!A+$Il(@lAOOhQAznqAyD|28p|1G3BkiU8wJf*0`jJHoIU1nYXEC94 z?u5|HS4hk7?NJJ&W~cbS_S;`_e{6W@sEpDX$wfv)(4G%{Cyn@8&K2pL*j~-hUbO19 zk&xAL`@UchwUC% zHxtwHrKIpGqTMjFr|%HN6s@4ibYTtm0H}m*k^|}R`)0E33L5u4BGR(Pq6(6PMT~jc z#9l`y6qk0l3SaQnx=%aS{~~AXt8VcVKIp;H<0#IcPt?9wemC_Go}$%m@2>0C8K+%C zP@STxzh!$!KXt$LgP2~elJDUv;0u1%Rhu41n&9S;5$5O?;G0YO){ot5_+syilDcb2 zM$hzK>ZCj03IU@_x{@Wx^MdxB$vj7)h|l1wPao&3C$>QJ5K_A=-@Z{ho=@Gvrdnkm z+FiGvRpxGqPxABFHOpGHw-Yh<{9mn{cT`i!9>*0{KoJm85s+e#<$)kYW0WF6B1lJR zQbH4u5C|m@x`?QR-j*%`iX>8kNDXaKdI?DANk9=2rAAOdK;_-IZ&&v`-#Poo`{$lJ zbEf=$bMN`io!`v;_yw0Kgoy6)R4T4YlJB|xdc3K=|6>(o@kEMW6eU}vvN$@fS_OBPq)iZ5`86Cr<#gnOh z@vbfeM2;+VW1AazU^WR`4b@e6eGTFKEiTy5LQ++-E>VLcfmSl;$FS>~qP;u8s3|%< zh7pjE6f1grYe!6W#c`i@SpYPhk&OxjxlVY$9Hn(nEs`@sj<3b`O+@WN>!U@A?~_In zULNeAsfly9`j$j^e~ga)sPZqhOg>r6xAFCdM*!L%Bt*$ZhGtDI@r~|7N&L46X&Q z|NFQTaNHq8GlcJ4sh1=oNcf&w~gh7Nw#`q3YVV*OOpt@*eaG zhl@eiZmV$^y((evFtnCu%%>Wd+?2T!Z$`}y(Nvu>%jl+AggJU6FK;5rWmfjC0u(I* zp;=URIGPI{iq{6CQ$6b?V>8@Zqx+M-2<9~?nlAO747E`H^i+B5+-$=eU72=YvX z#VQSrp)s8BK6y`cYmfg(kLd^l@h^EkB-oI>BTUPXxgjv0Gp@2ekULv}kL%VkxSBX= zSGH_la^VD-tn7Jm!oMSA{r#RWrlA-+ZrR*tWAdHJGorveF~&^eQIAh-p-3$R;-S`< z=4e~OBHF3iYshi1sutet<|8&zxxhRl6U3wIGRL@wW{R~STM$tt~`7(-+IqO|8G24gMiTc`^YqS36s(P=*twEz zeh(d&*+iEP>;||zJX>0rjdILK@-=rf-8iOuz?U9?&ulzi>npjnJ&M=t*PU8S5iiVN zGKH!;>$!EUi~!5~HwR8?~kT zdv9>%cCGtm&;h)gpxc@nbk$CNp>1X~DxOdGvJLJ`{v@?Y*Yd;BbLeu5^^SWNr(5S6 zm?}e8ZRiMb)590!1_lIr%Ep`AXN~t--jnB?b2CRFq4qLJBJhGt+@1Gd)i-Ng!;|=C zDx>ti6or&&lw)@6+~BzTYu>qAO7ppoG@g(GQh<{V@19QUmN$Eu^I9#}_<3Q9IAsnQ zt{lR~aFVZz9|m@EO2apPtNGmfRB!sx&`l`vY^^rgCZbBFjDGxy!4B4Z_2{{cpIjn;*813WbWq&44*u-??LdOfr-tljx)S|GXF6wFx;(RDe{U~! z>C~$U5WF$Jy{UqQZ0ce%Mtg0psvl8YB-n@#T7EjEs92AVaO|{$Pv-Wn#bz`-W}%a= zATqbF(Y^rEs-5N?DotglMQqVbXx#A#<5c&vbR9w?gkwPGF9W5vat_4LZPWuxBR z1kkcUJu!#c>I(=X>6ZsKLFsx8tBHvAvtu2@Huxk!2EA_Iq zZfvyJ+P0o%s&m`Y)K2Q4b5wXt14Gn}6#>WDu%G+lOW~)yw4~q%G9X1Mf+DpdoKABw zL#rD^3%`j#v~sq!N)pxYSJ7Bzm;>f#xv_hy@ypy_8H zCJ=pGf>!(MkdeTz>1U<##x!lX4Ph|4+N}2KFCcNWBtqlb7P9T55BG57wMzz03f@b$ zm(FKyXZj}A1dNW2)ykQCtxm@Y^Isv3i9*;a40b=Pgfb`zMQe$t_}($HJEwEsXqE;K z$7WEEE)j`63QD;Y@@;|SQl=?DtZSEnp!i@%Qc)Ke)?=dF6m7Y;nonytm4akG`ASW= zL27LKr)=j!qBAXCWAt84?R|iP!CZ11XKSQ|uzj_KzJ0V@zmoNq>i};S z^`TpXM^IRz&le8{gM*eSSqZk>lJ9UG$j5NNL7)y=pz#Cfx$A_`swbD5Rw9Qo8!An1 z$n=cTF2~qg7TSA_I|4n0#^QE8l+GdH?<)lm5fPY%cH1T7u)a?3)#kN}i{Yt{iIXyX z{S^kPJ)HdfX^tr&hMR2Dal|EUt+cWy^c;w1N@>|UPvhE<@&E?PAgI$HUGod*p)cXI z)Pc30Wa)i3P!6>BBss|9$tqePESgRkraI9c>t3*MRyr2tZZ!{{CI=V+_aI&;n(^Lx=YRk8F|#SK_TY)Pr|X!z+h%mk9UDmHVF`3Pp!wf zs}7oNeg^w{5&qR_4TSmZ*QuHq^_ zw5-|OcYtl}sCiN4w!2`bU>(NdlN{+VM8EP?Btm)s4ER1EhJ7H1_j!LPSbeQGV~m>D zw4pRzth3;$KnUQWEs{KijH@|KxxbZFNRbjHWZT?vL7+@~TsU_l^rS%#m6rxr%=D8a zXR^d+_oCygai2WtcLKF07X^%9W;M~3mJ%brFAXAEXoVxA-8$__2^9XFo8Fj+Q00iC&FM}YXMuk@SI$$YBSQv(vWI-em#M{;uPH>GU6x{Xc~Jzhcvw>p z#XRqyhSz?|r~Dp`V@(9#Q>@qYC#mmQ|DWQrlV5)a@VC*bE!-O5KsXB$t$WD`M!4v3 F``;!2^OgVr diff --git a/src/collections/_documentation/performance/distributed-tracing.md b/src/collections/_documentation/performance/distributed-tracing.md index 1dc49ef379ef7..6483f2076e6ec 100644 --- a/src/collections/_documentation/performance/distributed-tracing.md +++ b/src/collections/_documentation/performance/distributed-tracing.md @@ -44,11 +44,15 @@ Before learning how to enable tracing in your application, it helps to understan ### Traces, Transactions, and Spans -A **trace** represents the record of the entire operation you want to measure or track - like page load, an instance of a user completing some action in your application, or a cron job in your backend. When a trace includes work in multiple services, such as those listed above, it's called a **distributed trace**, because the trace is distributed across those services. To recap, a trace is a record of an operation; a distributed trace is a record of an operation that's distributed across services. +A **trace** represents the record of the entire operation you want to measure or track - like page load, an instance of a user completing some action in your application, or a cron job in your backend. When a trace includes work in multiple services, such as those listed above, it's called a **distributed trace**, because the trace is distributed across those services. -Each trace consists of one or more tree-like structures called **transactions**, the nodes of which are called **spans**. In most cases, each transaction represents a single instance of a service being called, and each span within that transaction represents that service performing a single unit of work, whether calling a function within that service or making a call to a different service. +Each trace consists of one or more tree-like structures called **transactions**, the nodes of which are called **spans**. In most cases, each transaction represents a single instance of a service being called, and each span within that transaction represents that service performing a single unit of work, whether calling a function within that service or making a call to a different service. Here's an example trace, broken down into transactions and spans: -Because a transaction has a tree structure, top-level spans can themselves be broken down into smaller spans, mirroring the way that one function may call a number of other, smaller functions; this is expressed using the parent-child metaphor, so that every span may be the **parent span** to multiple other spans. Further, since all trees must have a single root, one span in a transaction always represents the transaction itself, with all other spans in the transaction descending from that root span. +[{% asset performance/trace-transactions-spans-generic.png alt="Diagram illustrating how a trace is composed of multiple transactions, and each transaction is composed of multiple spans." %}]({% asset performance/trace-transactions-spans-generic.png @path %}) + +Because a transaction has a tree structure, top-level spans can themselves be broken down into smaller spans, mirroring the way that one function may call a number of other, smaller functions; this is expressed using the parent-child metaphor, so that every span may be the **parent span** to multiple other **child spans**. Further, since all trees must have a single root, one span in every transaction always represents the transaction itself, with all other spans in the transaction descending from that root span. Here's a zoomed-in view of one of the transactions from the diagram above: + +[{% asset performance/span-parent-child-relationship.png alt="Diagram illustrating the parent-child relationship between spans within a single transaction." %}]({% asset performance/span-parent-child-relationship.png @path %}) To make all of this more concrete, let's consider our example web app again. @@ -89,11 +93,15 @@ Each transaction would be broken down into **spans** as follows: - 1 span for the rendering task, which itself contains - 2 child spans, one for each JSON request -Let's pause here to make an important point: Some, though not all, of the browser transaction spans listed have a direct correspondence to backend transactions listed earlier. Specifically, each request _span_ in the browser transaction corresponds to a separate request _transaction_ in the backend. In this situation, when a span in one service gives rise to a transaction in a subsequent service, we call the original span a parent span to _both_ the transaction and its root span. +Let's pause here to make an important point: Some, though not all, of the spans listed here in the browser transaction have a direct correspondence to backend transactions listed earlier. Specifically, each request span in the browser transaction corresponds to a separate request transaction in the backend. In this situation, when a span in one service gives rise to a transaction in a subsequent service, we call the original span a parent span to _both_ the transaction and its root span. In the diagram below, the squggly lines represent this parent-child relationship. + +[{% asset performance/trace-transactions-spans-concrete-example.png alt="Diagram illustrating the trace-transaction-span relationship illustrated above, now applied to the example." %}]({% asset performance/trace-transactions-spans-concrete-example.png @path %}) -In our example, every transaction other than the initial browser page-load transaction is the child of a span in another service, which means that every root span other than the browser transaction root has a parent span (albeit in a different service). In a fully-instrumented system (one in which every service has tracing enabled) this pattern will always hold true. The only parentless span will be the root of the initial transaction; every other span will have a parent. Further, parents and children will always live in the same service, except in the case where the child span is the root of a child transaction, in which case the parent span will live in the calling service and the child transaction/child root span will live in the called service. +In our example, every transaction other than the initial browser page-load transaction is the child of a span in another service, which means that every root span other than the browser transaction root has a parent span (albeit in a different service). -Put another way, a fully-instrumented system creates a trace which is itself a connected tree - with each transaction a subtree - and in that tree, the boundaries between subtrees/transactions are precisely the boundaries between services. +In a fully-instrumented system (one in which every service has tracing enabled) this pattern will always hold true. The only parentless span will be the root of the initial transaction; every other span will have a parent. Further, parents and children will always live in the same service, except in the case where the child span is the root of a child transaction, in which case the parent span will live in the calling service and the child transaction/child root span will live in the called service. + +Put another way, a fully-instrumented system creates a trace which is itself a connected tree - with each transaction a subtree - and in that tree, the boundaries between subtrees/transactions are precisely the boundaries between services. The diagram above shows one branch of our example's full trace tree. Now, for the sake of completeness, back to our spans: @@ -115,9 +123,6 @@ Now, for the sake of completeness, back to our spans: - 1 span for the query retrieving data To wrap up the example: after instrumenting all of your services, you might discover that - for some reason - it's the auth query in your database server that is making things slow, accounting for more than half of the time it takes for your entire page load process to complete. Tracing can't tell you _why_ that's happening, but at least now you know where to look! - - ### Further Examples @@ -295,7 +300,7 @@ The following functions aggregate transaction counts and the rate at which trans - average requests (transactions) per second - average requests (transactions) per minute -Each of these functions is calculated with respect to the collection of transactions within the given row, which means the numbers will change as you filter your data or change the time window. Also, if you have set up your SDK to [sample your data](#data-sampling), remember that only the transactions that are sent to Sentry are counted. So if a row containing transactions representing requests to a given endpoint is calculated to be receiving 5 requests per second, and you've got a 25% sampling rate enabled, in reality you're getting approximately 20 requests to that endpoint each second. (20 because you're sampling 25% - or 1/4 - of your data, so your real volume is 4 times what you're seeing in Sentry.) +Each of these functions is calculated with respect to the collection of transactions within the given row, which means the numbers will change as you filter your data or change the time window. Also, if you have set up your SDK to [sample your data](#data-sampling), remember that only the transactions that are sent to Sentry are counted. So if a row containing transactions representing requests to a given endpoint is calculated to be receiving 5 requests per second, and you've got a 25% sampling rate enabled, in reality you're getting approximately 20 requests to that endpoint each second. (20 because you're collecting 25% - or 1/4 - of your data, so your real volume is 4 times what you're seeing in Sentry.) ### Transaction Detail View