Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion docs/observer.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# Incremental snapshots

After completing a full snapshot, we need to record events that change the state.

Right now, rrweb records the following events (we will expand upon this):
Expand All @@ -18,9 +19,11 @@ Right now, rrweb records the following events (we will expand upon this):
- Input

## Mutation Observer

Since we don't execute any JavaScript during replay, we instead need to record all changes scripts make to the document.

Consider this example:

> User clicks a button. A dropdown menu appears. User selects the first item. The dropdown menu disappears.

During replay, the dropdown menu does not automatically appear after the "click button" is executed, because the original JavaScript is not part of the recording. Thus, we need to record the creation of the dropdown menu DOM nodes, the selection of the first item, and subsequent deletion of the dropdown menu DOM nodes. This is the most difficult part.
Expand All @@ -33,9 +36,10 @@ The first thing to understand is that MutationObserver uses a **Bulk Asynchronou

This mechanism is not problematic for normal use, because we do not only have the mutation record, but we can also directly access the DOM object of the mutated node as well as any parent, child and sibling nodes.

However in rrweb, since we have a serialization process, we need more sophisticated soluation to be able to deal with various scenarios.
However in rrweb, since we have a serialization process, we need more sophisticated solution to be able to deal with various scenarios.

### Add node

For example, the following two operations generate the same DOM structure, but produce a different set of mutation records:

```
Expand Down Expand Up @@ -63,32 +67,40 @@ We already introduced in the [serialization design document](./serialization.md)
As you can see, since we have delayed serialization of the newly added nodes, all mutation records also need to be processed first, and only then the new nodes can be de-duplicated without causing trouble.

### Remove node

When processing mutation records, we may encounter a removed node that has not yet been serialized. That indicates that it is a newly added node, and the "add node" mutation record is also somewhere in the mutation records we received. We label these nodes as "dropped nodes".

There are two cases we need to handle here:

1. Since the node was removed already, there is no need to replay it, and thus we remove it from the newly added node pool.
2. This also applies to descendants of the dropped node, thus when processing newly added nodes we need to check if it has a dropped node as an ancestor.

### Attribute change

Although MutationObserver is an asynchronous batch callback, we can still assume that the time interval between mutations occurring in a callback is extremely short, so we can optimize the size of the incremental snapshot by overwriting some data when recording the DOM property changes.

For example, resizing a `<textarea>` will trigger a large number of mutation records with varying width and height properties. While a full record will make replay more realistic, it can also result in a large increase in the number of incremental snapshots. After making a trade-off, we think that only the final value of an attribute of the same node needs to be recorded in a single mutation callback, that is, each subsequent mutation record will overwrite the attribute change part of the mutation record that existing before the write.

## Mouse movement

By recording the mouse movement position, we can simulate the mouse movement trajectory during replay.

Try to ensure that the mouse moves smoothly during replay and also minimize the number of corresponding incremental snapshots, so we need to perform two layers of throttling while listening to mousemove. The first layer records the mouse coordinates at most once every 20 ms, the second layer transmits the mouse coordinate set at most once every 500 ms to ensure a single snapshot doesn't accumulate a lot of mouse position data and becomes too large.

### Time reversal

We record a timestamp when each incremental snapshot is generated so that during replay it can be applied at the correct time. However, due to the effect of throttling, the timestamps of the mouse movement corresponding to the incremental snapshot will be later than the actual recording time, so we need to record a negative time difference for correction and time calibration during replay.

## Input

We need to observe the input of the three elements `<input>`, `<textarea>`, `<select>`, including human input and programmatic changes.

### Human input

For human input, we mainly rely on listening to the input and change events. It is necessary to deduplicate different events triggered for the same the human input action. In addition, `<input type="radio" />` is also a special kind of control. If the multiple radio elements have the same name attribute, then when one is selected, the others will be reversed, but no event will be triggered on those others, so this needs to be handled separately.

### Programmatic changes

Setting the properties of these elements directly through the code will not trigger the MutationObserver. We can still achieve monitoring by hijacking the setter of the corresponding property. The sample code is as follows:

```typescript
Expand Down
2 changes: 1 addition & 1 deletion packages/rrweb/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@
"ignore-styles": "^5.0.1",
"inquirer": "^6.2.1",
"jest-snapshot": "^23.6.0",
"jsdom": "^16.6.0",
"jsdom": "^17.0.0",
"jsdom-global": "^3.0.2",
"mocha": "^5.2.0",
"prettier": "2.2.1",
Expand Down
54 changes: 54 additions & 0 deletions packages/rrweb/src/record/observer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -472,6 +472,23 @@ function initInputObserver(
};
}

function getNestedCSSRulePositions(rule: CSSStyleRule): number[] {
const positions: Array<number> = [];
function recurse(rule: CSSRule, pos: number[]) {
if (rule.parentRule instanceof CSSGroupingRule) {
const rules = Array.from((rule.parentRule as CSSGroupingRule).cssRules);
const index = rules.indexOf(rule);
pos.unshift(index);
} else {
const rules = Array.from(rule.parentStyleSheet!.cssRules);
const index = rules.indexOf(rule);
pos.unshift(index);
}
return pos;
}
return recurse(rule, positions);
}

function initStyleSheetObserver(
cb: styleSheetRuleCallback,
mirror: Mirror,
Expand Down Expand Up @@ -500,9 +517,46 @@ function initStyleSheetObserver(
return deleteRule.apply(this, arguments);
};

const groupingInsertRule = CSSGroupingRule.prototype.insertRule;
CSSGroupingRule.prototype.insertRule = function (
rule: string,
index?: number,
) {
const id = mirror.getId(this.parentStyleSheet.ownerNode as INode);
if (id !== -1) {
cb({
id,
adds: [
{
rule,
index: [
...getNestedCSSRulePositions(this),
index || 0, // defaults to 0
],
},
],
});
}
return groupingInsertRule.apply(this, arguments);
};

const groupingDeleteRule = CSSGroupingRule.prototype.deleteRule;
CSSGroupingRule.prototype.deleteRule = function (index: number) {
const id = mirror.getId(this.parentStyleSheet.ownerNode as INode);
if (id !== -1) {
cb({
id,
removes: [{ index: [...getNestedCSSRulePositions(this), index] }],
});
}
return groupingDeleteRule.apply(this, arguments);
};

return () => {
CSSStyleSheet.prototype.insertRule = insertRule;
CSSStyleSheet.prototype.deleteRule = deleteRule;
CSSGroupingRule.prototype.insertRule = groupingInsertRule;
CSSGroupingRule.prototype.deleteRule = groupingDeleteRule;
};
}

Expand Down
34 changes: 26 additions & 8 deletions packages/rrweb/src/replay/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ import {
StyleRuleType,
VirtualStyleRules,
VirtualStyleRulesMap,
getNestedRule,
} from './virtual-styles';

const SKIP_TIME_THRESHOLD = 10 * 1000;
Expand Down Expand Up @@ -978,19 +979,26 @@ export class Replayer {
d.adds.forEach(({ rule, index }) => {
if (styleSheet) {
try {
const _index =
if (Array.isArray(index)) {
const positions = [...index];
const insertAt = positions.pop();
const nestedRule = getNestedRule(
styleSheet.cssRules,
positions,
);
nestedRule.insertRule(rule, insertAt);
} else {
const _index =
index === undefined
? undefined
: Math.min(index, styleSheet.cssRules.length);
try {
styleSheet.insertRule(rule, _index);
} catch (e) {
/**
* sometimes we may capture rules with browser prefix
* insert rule with prefixs in other browsers may cause Error
*/
}
} catch (e) {
/**
* sometimes we may capture rules with browser prefix
* insert rule with prefixs in other browsers may cause Error
*/
/**
* accessing styleSheet rules may cause SecurityError
* for specific access control settings
Expand All @@ -1008,7 +1016,17 @@ export class Replayer {
rules?.push({ index, type: StyleRuleType.Remove });
} else {
try {
styleSheet?.deleteRule(index);
if (Array.isArray(index)) {
const positions = [...index];
const deleteAt = positions.pop();
const nestedRule = getNestedRule(
styleSheet!.cssRules,
positions,
);
nestedRule.deleteRule(deleteAt || 0);
} else {
styleSheet?.deleteRule(index);
}
} catch (e) {
/**
* same as insertRule
Expand Down
44 changes: 40 additions & 4 deletions packages/rrweb/src/replay/virtual-styles.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,11 @@ export enum StyleRuleType {
type InsertRule = {
cssText: string;
type: StyleRuleType.Insert;
index?: number;
index?: number | number[];
};
type RemoveRule = {
type: StyleRuleType.Remove;
index: number;
index: number | number[];
};
type SnapshotRule = {
type: StyleRuleType.Snapshot;
Expand All @@ -23,14 +23,40 @@ type SnapshotRule = {
export type VirtualStyleRules = Array<InsertRule | RemoveRule | SnapshotRule>;
export type VirtualStyleRulesMap = Map<INode, VirtualStyleRules>;

export function getNestedRule(
rules: CSSRuleList,
position: number[],
): CSSGroupingRule {
const rule = rules[position[0]] as CSSGroupingRule;
if (position.length === 1) {
return rule;
} else {
return getNestedRule(
((rule as CSSGroupingRule).cssRules[position[1]] as CSSGroupingRule)
.cssRules,
position.slice(2),
);
}
}

export function applyVirtualStyleRulesToNode(
storedRules: VirtualStyleRules,
styleNode: HTMLStyleElement,
) {
storedRules.forEach((rule) => {
if (rule.type === StyleRuleType.Insert) {
try {
styleNode.sheet?.insertRule(rule.cssText, rule.index);
if (Array.isArray(rule.index)) {
const positions = [...rule.index];
const insertAt = positions.pop();
const nestedRule = getNestedRule(
styleNode.sheet!.cssRules,
positions,
);
nestedRule.insertRule(rule.cssText, insertAt);
} else {
styleNode.sheet?.insertRule(rule.cssText, rule.index);
}
} catch (e) {
/**
* sometimes we may capture rules with browser prefix
Expand All @@ -39,7 +65,17 @@ export function applyVirtualStyleRulesToNode(
}
} else if (rule.type === StyleRuleType.Remove) {
try {
styleNode.sheet?.deleteRule(rule.index);
if (Array.isArray(rule.index)) {
const positions = [...rule.index];
const deleteAt = positions.pop();
const nestedRule = getNestedRule(
styleNode.sheet!.cssRules,
positions,
);
nestedRule.deleteRule(deleteAt || 0);
} else {
styleNode.sheet?.deleteRule(rule.index);
}
} catch (e) {
/**
* accessing styleSheet rules may cause SecurityError
Expand Down
6 changes: 3 additions & 3 deletions packages/rrweb/src/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -295,7 +295,7 @@ export type textMutation = {
};

export type styleAttributeValue = {
[key:string]: styleValueWithPriority | string | false;
[key: string]: styleValueWithPriority | string | false;
};

export type styleValueWithPriority = [string, string];
Expand Down Expand Up @@ -384,11 +384,11 @@ export type scrollCallback = (p: scrollPosition) => void;

export type styleSheetAddRule = {
rule: string;
index?: number;
index?: number | number[];
};

export type styleSheetDeleteRule = {
index: number;
index: number | number[];
};

export type styleSheetRuleParam = {
Expand Down
Loading