Skip to content

update documentations around test isolation #4797

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 24 commits into from
Oct 24, 2022
Merged
Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
06d0eab
CLOUD-295: Add Dashboard CTAs to "Why Cypress -> Features"
debrisapron Dec 21, 2021
6e67656
Address review feedback
debrisapron Aug 10, 2022
5cee97a
Address review feedback
debrisapron Aug 16, 2022
aff4a1c
Address review feedback
debrisapron Oct 18, 2022
f52dcc9
update experimental warning for session to reflect async/attach chang…
AtofStryker Oct 19, 2022
0d5121c
update documentations around test isolation
emilyrohrbough Oct 19, 2022
22ccb4d
Merge pull request #4284 from cypress-io/content/why-cypress-features…
debrisapron Oct 21, 2022
4b13469
Add note about git metadata issues to AWS CodeBuild page
debrisapron Oct 21, 2022
fd6448e
Merge pull request #4804 from cypress-io/content/emphasize-git-metada…
debrisapron Oct 21, 2022
5faab5c
Update content/api/commands/session.md
emilyrohrbough Oct 24, 2022
7942fd9
Update content/api/commands/session.md
emilyrohrbough Oct 24, 2022
fb81d64
Update content/guides/core-concepts/writing-and-organizing-tests.md
emilyrohrbough Oct 24, 2022
7c7a654
Apply suggestions from code review
emilyrohrbough Oct 24, 2022
87e71a8
additional doc changes
emilyrohrbough Oct 24, 2022
d81aa76
Merge branch 'change-test-isolation' of https://github.com/cypress-io…
emilyrohrbough Oct 24, 2022
7b17ab6
fix markdown
emilyrohrbough Oct 24, 2022
577c2c9
align with recent origin doc changes
emilyrohrbough Oct 24, 2022
a661acd
Merge branch 'master' into change-test-isolation
emilyrohrbough Oct 24, 2022
635bed0
remove legacy mode concept & flag test isolation as experimental
emilyrohrbough Oct 24, 2022
2f1a075
Update content/api/commands/session.md
emilyrohrbough Oct 24, 2022
dee911b
Update content/api/commands/session.md
emilyrohrbough Oct 24, 2022
0ea202e
Apply suggestions from code review
emilyrohrbough Oct 24, 2022
090f784
Update content/api/commands/session.md
emilyrohrbough Oct 24, 2022
9243bb9
Apply suggestions from code review
emilyrohrbough Oct 24, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 23 additions & 22 deletions content/api/commands/origin.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,29 +16,30 @@ limitation determined by standard web security features of the browser. The
<strong class="alert-header"><Icon name="exclamation-triangle"></Icon>
Experimental</strong>

The `cy.origin()` command is currently experimental and can be enabled by
setting
the [`experimentalSessionAndOrigin`](/guides/references/experiments) flag
to `true` in the Cypress config.
The `session` API is currently experimental, and can be enabled by setting the
[`experimentalSessionAndOrigin`](/guides/references/experiments) option to
`true` in the Cypress config.

Enabling this flag does the following:

- It adds the [`cy.session()`](/api/commands/session) and `cy.origin()`
commands, and [`Cypress.session`](/api/cypress-api/session) API.
- It adds the `cy.session()` and [`cy.origin()`](/api/commands/origin) commands,
and [`Cypress.session`](/api/cypress-api/session) API.
- It adds the following new behaviors (that will be the default in a future
major version release of Cypress) at the beginning of each test:
- The page is cleared (by setting it to `about:blank`).
- All active session data (cookies, `localStorage` and `sessionStorage`)
across all domains are cleared.
- It supersedes
the [`Cypress.Cookies.preserveOnce()`](/api/cypress-api/cookies#Preserve-Once) and
[`Cypress.Cookies.defaults()`](/api/cypress-api/cookies#Defaults) methods.
major update of Cypress) at the beginning of each test:
- The
[`testIsolation`](/guides/core-concepts/writing-and-organizing-tests#Test-Isolation)
mode is enhanced from `legacy` mode to `on` mode such that
- The page is cleared (by setting it to `about:blank`).
- Cookies, local storage and session storage in all domains are cleared.
- It supersedes the
[`Cypress.Cookies.preserveOnce()`](/api/cypress-api/cookies#Preserve-Once) and
[`Cypress.Cookies.defaults()`](/api/cypress-api/cookies#Defaults) methods.
- Cross-origin requests will now succeed, however, to interact with a
cross-origin page you must use a `cy.origin` block.

Because the page is cleared before each
test, [`cy.visit()`](/api/commands/visit) must be explicitly called in each test
to visit a page in your application.
Because the page is cleared at the beginning of each test by default,
[`cy.visit()`](/api/commands/visit) must be explicitly called at the beginning
of each test.

</Alert>

Expand Down Expand Up @@ -390,12 +391,12 @@ and reuse it across tests.

Enabling the `experimentalSessionAndOrigin` flag makes the test-runner work
slightly differently, and some test suites that rely on the existing behaviour
may have to be updated. The most important of these changes is **test
isolation**. This means that after every test, the current page is reset to
`about:blank` and all active session data
(cookies, `localStorage` and `sessionStorage`) across all domains are cleared.
This change is opt-in for now, but will be standardized in a future major
release of Cypress, so eventually all tests will need to be isolated.
may have to be updated. The most important of these changes is
[**test isolation**]() which defaults to `on`. This means that after every test,
the current page is reset to `about:blank` and cookies, local storage and
session storage in all domains are cleared before each test. This change is
opt-in for now, but will be standardized in a future major release of Cypress,
so eventually all tests will need to be isolated.

Before this change, it was possible to write tests such that you could, for
example, log in to a CMS in the first test, change some content in the second
Expand Down
100 changes: 65 additions & 35 deletions content/api/commands/session.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,13 @@ Cache and restore [cookies](/api/cypress-api/cookies),
[`localStorage`](https://developer.mozilla.org/en-US/docs/Web/API/Window/localStorage),
and
[`sessionStorage`](https://developer.mozilla.org/en-US/docs/Web/API/Window/sessionStorage)
in order to reduce test setup times.
(i.e. session data) in order to recreate a consistent browser context between
tests.

The `cy.session()` command will inherent the
[`testIsolation`](/guides/core-concepts/writing-and-organizing-tests#Test-Isolation)
mode` value to determine where or not the page is cleared when cacheing and
Copy link
Contributor

@ryanthemanuel ryanthemanuel Oct 24, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
mode` value to determine where or not the page is cleared when cacheing and
mode value to determine where or not the page is cleared when caching and

restoring the browser context.

<Alert type="warning">

Expand All @@ -24,19 +30,18 @@ Enabling this flag does the following:
and [`Cypress.session`](/api/cypress-api/session) API.
- It adds the following new behaviors (that will be the default in a future
major update of Cypress) at the beginning of each test:
- The page is cleared (by setting it to `about:blank`). Disable this by
setting
[`testIsolation=legacy`](/guides/core-concepts/writing-and-organizing-tests#Test-Isolation).
- All active session data (cookies, `localStorage` and `sessionStorage`)
across all domains are cleared.
- It overrides the
[`Cypress.Cookies.preserveOnce()`](/api/cypress-api/cookies#Preserve-Once) and
- The
[`testIsolation`](/guides/core-concepts/writing-and-organizing-tests#Test-Isolation)
mode is enhanced from `legacy` mode to `on` mode such that:
- The page is cleared (by setting it to `about:blank`).
- Cookies, local storage and session storage in all domains are cleared.
- It supersedes
the [`Cypress.Cookies.preserveOnce()`](/api/cypress-api/cookies#Preserve-Once) and
[`Cypress.Cookies.defaults()`](/api/cypress-api/cookies#Defaults) methods.
- Cross-origin navigation will no longer fail immediately, but instead, time out
based on [`pageLoadTimeout`](/guides/references/configuration#Timeouts).
- Tests will no longer wait on page loads before moving on to the next test.
- Cross-origin requests will now succeed, however, to interact with a
cross-origin page you must use a `cy.origin` block.

Because the page is cleared at the beginning of each test,
Because the page is cleared at the beginning of each test by default,
[`cy.visit()`](/api/commands/visit) must be explicitly called at the beginning
of each test.

Expand Down Expand Up @@ -82,7 +87,7 @@ cy.session(name, () => {
// be done inside the setup function
cy.visit('/login')
cy.session(name, () => {
// need to call cy.visit() here because the page is blank when
// need to call cy.visit() here because the page is blank when
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// need to call cy.visit() here because the page is blank when
// need to call cy.visit() here because the page is blank when

// the setup function runs
cy.get('[data-test=name]').type(name)
cy.get('[data-test=password]').type('s3cr3t')
Expand Down Expand Up @@ -129,8 +134,11 @@ runs, Cypress will preserve all cookies, `sessionStorage`, and `localStorage`,
so that subsequent calls to `cy.session()` with the same `id` will bypass
`setup` and just restore the cached session data.

The page and all active session data (cookies, `localStorage` and
`sessionStorage`) across all domains are always cleared before `setup` runs.
The page is cleared before `setup` when `testIsolation='on'` and is not cleared
when `testIsolation='off'`.

Cookies, local storage and session storage in all domains are always cleared
before `setup` runs, regardless of the testIsolation configuration.

**<Icon name="angle-right"></Icon> options** **_(Object)_**

Expand Down Expand Up @@ -332,10 +340,10 @@ describe('account details', () => {

### Switching sessions inside tests

Because `cy.session()` clears the page and all active session data before
running `setup`, you can use it to easily switch between sessions without first
needing to log the previous user out. This allows tests to more accurately
represent real-world scenarios and helps keep test run times short.
Because `cy.session()` clears the page and all session data before running
`setup`, you can use it to easily switch between sessions without first needing
to log the previous user out. This allows tests to more accurately represent
real-world scenarios and helps keep test run times short.

```jsx
const login = (name) => {
Expand Down Expand Up @@ -670,24 +678,46 @@ it('t3', () => {

## Notes

### When the page and active session data are cleared
### When the page and session data are cleared

### Test Isolation `on`

The page is cleared and cookies, local storage and session storage (session
data) in all domains are cleared automatically when `cy.session()` runs and
`testIsolation` is `on`. This guarantees consistent behavior whether a session
is being created or restored and allows you to switch sessions without first
having to explicitly log out.

| | Page cleared (test) | Session data cleared |
| -------------------------- | :---------------------------------------------: | :---------------------------------------------: |
| Before `setup` | <Icon name="check-circle" color="green"></Icon> | <Icon name="check-circle" color="green"></Icon> |
| Before `validate` | <Icon name="check-circle" color="green"></Icon> | |
| Before `cy.session()` ends | <Icon name="check-circle" color="green"></Icon> | |

[`cy.visit()`](/api/commands/visit) must be explicitly called afterwards to
ensure the page to test is loaded.

### Test Isolation `iff`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
### Test Isolation `iff`
### Test Isolation `off`


When `testIsolation` is `off`, the page will not clear, however, the session
data will clear when `cy.session()` runs.

| | Page cleared (test) | Session data cleared |
| -------------------------- | :-----------------: | :---------------------------------------------: |
| Before `setup` | | <Icon name="check-circle" color="green"></Icon> |
| Before `validate` | | |
| Before `cy.session()` ends | | |

The page is cleared and all active session data (cookies, `localStorage`, and
`sessionStorage`) across all domains are cleared automatically when
`cy.session()` runs. This guarantees consistent behavior whether a session is
being created or restored and allows you to switch sessions without first having
to explicitly log out.
[`cy.visit()`](/api/commands/visit) does not need to be called afterwards to
ensure the page to test is loaded.

| | Current page cleared | Active session data cleared |
| -------------------- | :---------------------------------------------: | :---------------------------------------------: |
| Before `setup` | <Icon name="check-circle" color="green"></Icon> | <Icon name="check-circle" color="green"></Icon> |
| Before `validate` | <Icon name="check-circle" color="green"></Icon> | |
| After `cy.session()` | <Icon name="check-circle" color="green"></Icon> | |
NOTE: Turning test isolation off may improve performance of end-to-end tests,
however, previous tests could impact the browser state of the next test and
cause inconsistency when using .only(). Be mindful to write isolated tests when
test isolation is off.

Calling `cy.session()` clears the current page in addition to restoring the
cached session data. [`cy.visit()`](/api/commands/visit) must be explicitly
called afterwards to ensure a page is visited if you did not provide a
`validate` function that called `cy.visit()`.
When test isolation is `off`, it is encouraged to setup your session in a before
hook or in the first test to ensure a clean setup.

### Session caching

Expand Down Expand Up @@ -881,7 +911,7 @@ that were run when creating and/or validating the session.
In this image, a saved session is restored, but when `/personal` is visited in
the `validate` function, the app redirects to `/signin`, which invalidates the
session. A new session is created by visiting `/signin` where the user is logged
in, after which, validation succeeds, and the session is made active for the
in, after which, validation succeeds, and the session is available for the
remainder of the test.

<DocsImage src="/img/api/session/session-expanded.png" alt="Recreated session (expanded)"></DocsImage>
Expand Down
27 changes: 15 additions & 12 deletions content/api/cypress-api/session.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,26 +11,29 @@ be used alongside the [`cy.session()`](/api/commands/session) command.
Experimental</strong>

The `session` API is currently experimental, and can be enabled by setting the
[`experimentalSessionSupport`](/guides/references/experiments) flag to `true` in
the Cypress config or by using [`Cypress.config()`](/api/cypress-api/config) at
the top of a spec file.
[`experimentalSessionAndOrigin`](/guides/references/experiments) option to
`true` in the Cypress config.

Enabling this flag does the following:

- It adds the [`cy.session()`](/api/commands/session) command for use in tests.
- It adds the [`Cypress.session`](/api/cypress-api/session) API.
- It adds the `cy.session()` and [`cy.origin()`](/api/commands/origin) commands,
and [`Cypress.session`](/api/cypress-api/session) API.
- It adds the following new behaviors (that will be the default in a future
major update of Cypress) at the beginning of each test:
- The page is cleared (by setting it to `about:blank`).
- All active session data (cookies, `localStorage` and `sessionStorage`)
across all domains are cleared.
- It overrides the
- The
[`testIsolation`](/guides/core-concepts/writing-and-organizing-tests#Test-Isolation)
mode is enhanced from `legacy` mode to `on` mode such that
- The page is cleared (by setting it to `about:blank`).
- Cookies, local storage and session storage in all domains are cleared.
- It supersedes the
[`Cypress.Cookies.preserveOnce()`](/api/cypress-api/cookies#Preserve-Once) and
[`Cypress.Cookies.defaults()`](/api/cypress-api/cookies#Defaults) methods.
- Cross-origin requests will now succeed, however, to interact with a
cross-origin page you must use a `cy.origin` block.

Because the page is cleared before each test,
[`cy.visit()`](/api/commands/visit) must be explicitly called in each test to
visit a page in your application.
Because the page is cleared at the beginning of each test by default,
[`cy.visit()`](/api/commands/visit) must be explicitly called at the beginning
of each test.

</Alert>

Expand Down
15 changes: 15 additions & 0 deletions content/guides/continuous-integration/aws-codebuild.md
Original file line number Diff line number Diff line change
Expand Up @@ -413,6 +413,21 @@ phases:

## Using the Cypress Dashboard with AWS CodeBuild

<Alert type="warning">

Dashboard analytics are dependent on your CI environment reliably providing
commit SHA data (typically via an environment variable) which is not always
present by default. This is not a problem for most users, but if you are facing
integration issues with your CodeBuild setup, please make sure the git
information is being sent properly by following
[these guidelines](/guides/continuous-integration/introduction#Git-information),
or just see
[the example `codebuild.yml` file at the top of this page](#Basic-Setup). If you
are still facing issues after this, please
[contact us](mailto:[email protected]).

</Alert>

In the AWS CodeBuild configuration we have defined in the previous section, we
are leveraging three useful features of the
[Cypress Dashboard](https://on.cypress.io/dashboard):
Expand Down
Loading