Skip to content

Commit b83a6ac

Browse files
emilyrohrboughchrisbreidingmjhenkesAtofStryker
authored
12: update test isolation docs to use true/false instead of on/off (#4890)
Co-authored-by: Chris Breiding <[email protected]> Co-authored-by: Matt Henkes <[email protected]> Co-authored-by: Bill Glesias <[email protected]>
1 parent 037cf39 commit b83a6ac

File tree

6 files changed

+182
-127
lines changed

6 files changed

+182
-127
lines changed

content/_data/sidebar.json

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -197,6 +197,10 @@
197197
"title": "Writing and Organizing Tests",
198198
"slug": "writing-and-organizing-tests"
199199
},
200+
{
201+
"title": "Test Isolation",
202+
"slug": "test-isolation"
203+
},
200204
{
201205
"title": "Retry-ability",
202206
"slug": "retry-ability"

content/api/commands/session.md

Lines changed: 21 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,8 @@ tests.
1212

1313
The `cy.session()` command will inherit the
1414
[`testIsolation`](/guides/core-concepts/writing-and-organizing-tests#Test-Isolation)
15-
mode value to determine whether or not the page is cleared when caching and
16-
restoring the browser context.
15+
value to determine whether or not the page is cleared when caching and restoring
16+
the browser context.
1717

1818
## Syntax
1919

@@ -102,11 +102,11 @@ runs, Cypress will preserve all cookies, `sessionStorage`, and `localStorage`,
102102
so that subsequent calls to `cy.session()` with the same `id` will bypass
103103
`setup` and just restore the cached session data.
104104

105-
The page is cleared before `setup` when `testIsolation='on'` and is not cleared
106-
when `testIsolation='off'`.
105+
The page is cleared before `setup` when `testIsolation` is enabled and is not
106+
cleared when `testIsolation` is disabled.
107107

108108
Cookies, local storage and session storage in all domains are always cleared
109-
before `setup` runs, regardless of the testIsolation configuration.
109+
before `setup` runs, regardless of the `testIsolation` configuration.
110110

111111
**<Icon name="angle-right"></Icon> options** **_(Object)_**
112112

@@ -645,13 +645,13 @@ it('t3', () => {
645645

646646
### When the page and session data are cleared
647647

648-
### Test Isolation `on`
648+
### Test Isolation Enabled
649649

650650
The page is cleared and cookies, local storage and session storage (session
651-
data) in all domains are cleared automatically when `cy.session()` runs and
652-
`testIsolation` is `on`. This guarantees consistent behavior whether a session
653-
is being created or restored and allows you to switch sessions without first
654-
having to explicitly log out.
651+
data) in all domains are cleared automatically when `cy.session()` runs and test
652+
isolation is enabled with `testIsolation=true`, This guarantees consistent
653+
behavior whether a session is being created or restored and allows you to switch
654+
sessions without first having to explicitly log out.
655655

656656
| | Page cleared (test) | Session data cleared |
657657
| -------------------------- | :---------------------------------------------: | :---------------------------------------------: |
@@ -661,10 +661,10 @@ having to explicitly log out.
661661
[`cy.visit()`](/api/commands/visit) must be explicitly called afterwards to
662662
ensure the page to test is loaded.
663663

664-
### Test Isolation `off`
664+
### Test Isolation Disabled
665665

666-
When `testIsolation` is `off`, the page will not clear, however, the session
667-
data will clear when `cy.session()` runs.
666+
When test isolation is disabled with `testIsolation=false`, the page will not
667+
clear, however, the session data will clear when `cy.session()` runs.
668668

669669
| | Page cleared (test) | Session data cleared |
670670
| -------------------------- | :-----------------: | :---------------------------------------------: |
@@ -674,13 +674,13 @@ data will clear when `cy.session()` runs.
674674
[`cy.visit()`](/api/commands/visit) does not need to be called afterwards to
675675
ensure the page to test is loaded.
676676

677-
NOTE: Turning test isolation off may improve performance of end-to-end tests,
677+
NOTE: Disabling test isolation may improve performance of end-to-end tests,
678678
however, previous tests could impact the browser state of the next test and
679679
cause inconsistency when using .only(). Be mindful to write isolated tests when
680-
test isolation is off.
680+
test isolation is disabled.
681681

682-
When test isolation is `off`, it is encouraged to setup your session in a before
683-
hook or in the first test to ensure a clean setup.
682+
When test isolation is disabled, it is encouraged to setup your session in a
683+
before hook or in the first test to ensure a clean setup.
684684

685685
### Session caching
686686

@@ -823,9 +823,10 @@ generate random unique ids if an arbitrary name-space does not meet your needs.
823823

824824
#### Why are all my Cypress commands failing after calling `cy.session()`?
825825

826-
When test isolation is `on`, ensure that you're calling
827-
[`cy.visit()`](/api/commands/visit) after calling `cy.session()`, otherwise your
828-
tests will be running on a blank page.
826+
When
827+
[`testIsolation`](/guides/core-concepts/writing-and-organizing-tests#Test-Isolation)
828+
is enabled, ensure that you're calling [`cy.visit()`](/api/commands/visit) after
829+
calling `cy.session()`, otherwise your tests will be running on a blank page.
829830

830831
#### Why am I seeing `401` errors after calling `cy.session()`?
831832

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
---
2+
title: 'Test Isolation'
3+
---
4+
5+
<Alert type="info">
6+
7+
## <Icon name="graduation-cap"></Icon> What you'll learn
8+
9+
- What is test isolation
10+
- How it impacts E2E Testing vs Component Testing
11+
- Test isolation trade-offs
12+
13+
</Alert>
14+
15+
## What is Test Isolation?
16+
17+
<Alert type="success">
18+
19+
<Icon name="check-circle" color="green"></Icon> **Best Practice:** Tests should
20+
always be able to be run independently from one another **and still pass**.
21+
22+
</Alert>
23+
24+
As stated in our mission, we hold ourselves accountable to champion a testing
25+
process that actually works, and have built Cypress to guide developers towards
26+
writing independent tests from the start.
27+
28+
We do this by cleaning up state _before_ each test to ensure that the operation
29+
of one test does not affect another test later on. The goal for each test should
30+
be to **reliably pass** whether run in isolation or consecutively with other
31+
tests. Having tests that depend on the state of an earlier test can potentially
32+
cause nondeterministic test failures which make debugging challenging.
33+
34+
Cypress will start each test with a clean test slate by restoring and clearing
35+
all:
36+
37+
- [aliases](/api/commands/as)
38+
- [clock mocks](/api/commands/clock)
39+
- [intercepts](/api/commands/intercept)
40+
- [routes](/api/commands/route)
41+
- [spies](/api/commands/spy)
42+
- [stubs](/api/commands/stub)
43+
- [viewport changes](/api/commands/viewport)
44+
45+
In additional to a clean test slate, Cypress also believes in running tests in a
46+
clean browser context such that the application or component under test behaves
47+
consistently when ran. This behavior is described as `testIsolation`.
48+
49+
The test isolation is a global configuration and can be overridden for
50+
end-to-end testing at the `describe` level with the
51+
[`testIsolation`](/guides/references/configuration#Global) option.
52+
53+
## Test Isolation in End-to-End Testing
54+
55+
Cypress supports enabling or disabling test isolation in end-to-end testing to
56+
describe if a suite of tests should run in a clean browser context or not.
57+
58+
### Test Isolation Enabled
59+
60+
When test isolation is enabled, Cypress resets the browser context _before_ each
61+
test by:
62+
63+
- clearing the dom state by visiting `about:blank`
64+
- clearing [cookies](/api/cypress-api/cookies) in all domains
65+
- clearing
66+
[`localStorage`](https://developer.mozilla.org/en-US/docs/Web/API/Window/localStorage)
67+
in all domains
68+
- clearing
69+
[`sessionStorage`](https://developer.mozilla.org/en-US/docs/Web/API/Window/sessionStorage)
70+
in all domains
71+
72+
Because the test starts in a fresh browser context, you must re-visit your
73+
application and perform the series of interactions needed to build the dom and
74+
browser state for each test.
75+
76+
Additionally, the [`cy.session()`](/api/commands/session) command will inherent
77+
this configuration and will clear the page and current browser context when
78+
establishing a browser session. This is so tests can reliably pass when run
79+
standalone or in a randomized order.
80+
81+
### Test Isolation Disabled
82+
83+
When test isolation is disabled, Cypress will not alter the browser context
84+
before the test starts. The page does not clear between tests and cookies, local
85+
storage and session storage will be available across tests in that suite.
86+
Additionally, the [`cy.session()`](/api/commands/session) command will only
87+
clear the current browser context when establishing the browser session - the
88+
current page is not cleared.
89+
90+
### Quick Comparison
91+
92+
| testIsolation | beforeEach test | cy.session() |
93+
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ |
94+
| `true` | - clears page by visiting `about:blank`<br>- clears cookies in all domains<br>- local storage in all domains<br>- session storage in all domains | - clears page by visiting `about:blank`<br>- clears cookies in all domains<br>- local storage in all domains<br>- session storage in all domains |
95+
| `false` | does not alter the current browser context | <br>- clears cookies in all domains<br>- local storage in all domains<br>- session storage in all domains |
96+
97+
## Test Isolation in Component Testing
98+
99+
Cypress does not support configuring the test isolation behavior in component
100+
testing.
101+
102+
When running component tests, Cypress always resets the browser context _before_
103+
each test by:
104+
105+
- unmounting the rendered component under test
106+
- clearing [cookies](/api/cypress-api/cookies) in all domains
107+
- clearing
108+
[`localStorage`](https://developer.mozilla.org/en-US/docs/Web/API/Window/localStorage)
109+
in all domains
110+
- clearing
111+
[`sessionStorage`](https://developer.mozilla.org/en-US/docs/Web/API/Window/sessionStorage)
112+
in all domains
113+
114+
## Test Isolation Trade-offs
115+
116+
It is important to note that disabling test isolation may improve the overall
117+
performance of end-to-end tests, however, it can also cause state to "leak"
118+
between tests. This can make later tests dependent on the results of earlier
119+
tests, and potentially cause misleading test failures. It is important to be
120+
extremely mindful of how tests are written when using this mode, and ensure that
121+
tests continue to run independently of one another.
122+
123+
The best way to ensure your tests are independent is to add a `.only()` to your
124+
test and verify it can run successfully without the test before it.

content/guides/core-concepts/writing-and-organizing-tests.md

Lines changed: 16 additions & 89 deletions
Original file line numberDiff line numberDiff line change
@@ -575,95 +575,22 @@ As stated in our mission, we hold ourselves accountable to champion a testing
575575
process that actually works, and have built Cypress to guide developers towards
576576
writing independent tests from the start.
577577

578-
We do this by cleaning up state _before_ each test to ensure that the operation
579-
of one test does not affect another test later on. The goal for each test should
580-
be to **reliably pass** whether run in isolation or consecutively with other
581-
tests. Having tests that depend on the state of an earlier test can potentially
582-
cause nondeterministic test failures which makes debugging challenging.
583-
584-
Cypress will start each test with a clean test slate by restoring and clearing
585-
all:
586-
587-
- [aliases](/api/commands/as)
588-
- [clock mocks](/api/commands/clock)
589-
- [intercepts](/api/commands/intercept)
590-
- [routes](/api/commands/route)
591-
- [spies](/api/commands/spy)
592-
- [stubs](/api/commands/stub)
593-
- [viewport changes](/api/commands/viewport)
594-
595-
In additional to a clean test slate, Cypress also believes in running tests in a
596-
clean browser context such that the application or component under test behaves
597-
consistently when ran. This concept is described as `testIsolation`.
598-
599-
The test isolation mode is a global configuration and can be overridden at the
600-
`describe` level with the
601-
[`testIsolation`](/guides/references/configuration#Global) option.
602-
603-
#### End-to-end testing
604-
605-
Cypress supports the following modes of test isolation in end-to-end testing to
606-
describe if a suite of tests should run in a clean browser context or not: `on`
607-
and `off`.
608-
609-
###### On Mode
610-
611-
This is the default test isolation behavior in Cypress. When in `on` mode,
612-
Cypress resets the browser context _before_ each test by:
613-
614-
- clearing the dom state by visiting `about:blank`
615-
- clearing [cookies](/api/cypress-api/cookies) in all domains
616-
- clearing
617-
[`localStorage`](https://developer.mozilla.org/en-US/docs/Web/API/Window/localStorage)
618-
in all domains
619-
- clearing
620-
[`sessionStorage`](https://developer.mozilla.org/en-US/docs/Web/API/Window/sessionStorage)
621-
in all domains
622-
623-
Because the test starts in a fresh browser context, you must re-visit your
624-
application and perform the series of interactions needed to build the dom and
625-
browser state for each test.
626-
627-
Additionally, the `cy.session()` command will inherent this mode and will clear
628-
the page and current browser context when establishing a browser session. This
629-
is so tests can reliably pass when run standalone or in a randomized order.
630-
631-
###### Off Mode
632-
633-
When in `off` mode, Cypress will not alter the browser context before the test
634-
starts. The page does not clear between tests and cookies, local storage and
635-
session storage will be available across tests in that suite. Additionally, the
636-
`cy.session()` command will only clear the current browser context when
637-
establishing the browser session - the current page will not clear.
638-
639-
It is important to note that while turning test isolation `off` may improve the
640-
overall performance of end-to-end tests, it can however cause state to "leak"
641-
between tests. This can make later tests dependent on the results of earlier
642-
tests, and potentially cause misleading test failures. It is important to be
643-
extremely mindful of how tests are written when using this mode, and ensure that
644-
tests continue to run independently of one another.
645-
646-
###### Mode Comparison
647-
648-
| testIsolation | beforeEach test | cy.session() |
649-
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ |
650-
| `on` | - clears page by visiting `about:blank`<br>- clears cookies in all domains<br>- local storage in all domains<br>- session storage in all domains | - clears page by visiting `about:blank`<br>- clears cookies in all domains<br>- local storage in all domains<br>- session storage in all domains |
651-
| `off` | does not alter the current browser context | <br>- clears cookies in all domains<br>- local storage in all domains<br>- session storage in all domains |
652-
653-
##### Component testing
654-
655-
Cypress only support testIsolation `on` in component testing.
656-
657-
When running component tests, Cypress resets the browser context _before_ each
658-
test by:
659-
660-
- clearing the page
661-
- clearing [cookies](/api/cypress-api/cookies)
662-
- clearing
663-
[`localStorage`](https://developer.mozilla.org/en-US/docs/Web/API/Window/localStorage)
664-
- clearing
665-
[`sessionStorage`](https://developer.mozilla.org/en-US/docs/Web/API/Window/sessionStorage)
666-
in all domains
578+
We do this by cleaning up test state and the browser context _before_ each test
579+
to ensure that the operation of one test does not affect another test later on.
580+
The goal for each test should be to **reliably pass** whether run in isolation
581+
or consecutively with other tests. Having tests that depend on the state of an
582+
earlier test can potentially cause nondeterministic test failures which makes
583+
debugging challenging.
584+
585+
The behavior of running tests in a clean browser context is described as
586+
`testIsolation`.
587+
588+
The test isolation is a global configuration and can be overridden for
589+
end-to-end testing at the `describe` level with the
590+
[`testIsolation`](/guides/references/configuration#e2e) option.
591+
592+
To learn more about this behavior and the trade-offs of disabling it, review our
593+
[Test Isolation guide](/guides/core-concepts/test-isolation).
667594

668595
### Test Configuration
669596

content/guides/references/configuration.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -208,7 +208,7 @@ object:
208208
| `specPattern` | `cypress/e2e/**/*.cy.{js,jsx,ts,tsx}` | A String or Array of glob patterns of the test files to load. |
209209
| `excludeSpecPattern` | `*.hot-update.js` | A String or Array of glob patterns used to ignore test files that would otherwise be shown in your list of tests. [Please read the notes on using this.](#excludeSpecPattern) |
210210
| `slowTestThreshold` | `10000` | Time, in milliseconds, to consider a test "slow" during `cypress run`. A slow test will display in orange text in the default reporter. |
211-
| `testIsolation` | `on` | The [test isolation level](/guides/core-concepts/writing-and-organizing-tests#Test-Isolation) applied to ensure a clean slate between tests. Options are `on` or `off`. |
211+
| `testIsolation` | `true` | Whether or not [test isolation](/guides/core-concepts/writing-and-organizing-tests#Test-Isolation) is enabled to ensure a clean browser context between tests. |
212212
| Option | Default | Description |
213213
| `experimentalRunAllSpecs` | `false` | Enables the "Run All Specs" UI feature, allowing the execution of multiple specs sequentially. |
214214

0 commit comments

Comments
 (0)