[Epic] Replay: Connect backend errors to Replays

[bruno-garcia]

Session Replay can give the reproduction steps users need to solve problems across their stack. This means not only frontend but also errors happening on the backend.

We want to better connect Session Replay with errors happening on the backend. So to help backend engineers solve bugs faster.

Phases

1. Propagate the replayId through traces. Today that requires performance monitoring enabled.

This will allow us to work on the product and CTAs. But limits the impact by only customers that have performance monitoring. We'll ideally achieve this without SDK change.

2. Decouple trace propagation from performance monitoring.

This will allow "all" backend errors to get linked to Replays, regardless of users having performance monitoring enabled.
For this to happen we'll need to change all Sentry SDKs. It'll affect other parts of the product like errors and require more coordination with other teams.

## Internal Release
- [x] RFC to propagate the replayId https://github.com/getsentry/rfcs/pull/60
- [x] Show the Replay CTA on issue details page for backend events
- [x] Replay count on the issue should reflect all replays connected to that backend issue
- [x] [Design] Replay Details should show count for both FE and BE errors
- [x] Display `Replay (count)` tab on issue details for backend projects

## EA Release
- [ ] https://github.com/getsentry/team-replay/issues/54

## GA Release
- [ ] https://github.com/getsentry/sentry/issues/46612
- [ ] https://github.com/getsentry/sentry/issues/48249
- [ ] https://github.com/getsentry/team-replay/issues/70
- [ ] https://github.com/getsentry/sentry-docs/issues/6890

(twitter)

[bruno-garcia]

Pulling an update from the RFC, from @JoshFerge

After doing some prototyping, I've found that each individual backend SDK will have to be modified in order to set the replay_id on the event. this is because the dynamic sampling context is not sent along to ingest on error events.

I also found that at least on the python SDK, the dynamic sampling values are hardcoded, which means that we'll have to update SDKs to add another value.

I'm curious because from reading the spec, it seems like we wanted more flexibility in the propagation.

Being able to simply copy key-value pairs from the baggage header onto the trace envelope header gives us the flexibility to provide dedicated API methods to propagate additional values using Dynamic Sampling Context. This, in return, allows users to define their own values in the Dynamic Sampling Context so they can sample by those in the Sentry interface.

Given these details, I believe we can take the following course of action to accomplish the initial goal of backend errors being tagged w/ replay_id when tracing is enabled:

• Modify the sentry-javascript replay SDK to expose the current replayId on the integration
• Modify the sentry-javascript DSC function to grab the replayId on the replay integration if it exists, and add it to the DSC obj.
• For each Backend SDK:
  • let replay_id propagate on the DSC
  • Set the replay_id on the scope like we do for trace_id currently
  • When we send an event, enrich it with the scope's replay_id
  • TODO: prioritize SDK implementations by adoption/importance
• From there, we shouldn't need any changes on relay
• Given this needs and SDK upgrade, we'd likely want to put SDK CTA upgrades for this feature in non-intrusive manner in the replays parts of the product.

This would allow users to navigate to replays from backend errors / issues. The extra work I described in option A of the RFC would allow users to search for replays by a specific backend error, but I don't think that's where the majority of the value lies in this integration, so for the initial feature release, I believe the above is the minimum set we'd need.

Given the amount of SDK work and friction in getting users to upgrade their SDKs, I do think it would be worthwhile to prototype what some kind of query engine for linking all of this via trace_id instead (Option D in the RFC). I will plan to spend a couple of days on this to explore feasibility before moving forward with the implementation / delegation of these SDK changes.

[JoshFerge]

update: going to write some quick thoughts on trying to do this backend only, but its likely not a path we're going to pursue, and will commence work on the replay_id trace propagation project shortly.