[test-runner] Advanced Concurrency Settings

[Westbrook]

When running tests in Firefox that rely on focus being present in the browser, running at a concurrency above 1 can create a situation wherein the browser/tab of the test is not currently in focus and the assertions therein will fail (rather flakily, too).

Thinking through possible approaches to the situation:

• Create concurrency settings by that apply by the browser rather than by an entire test run.
• Set/document a practice to separate focus relevant tests from other tests and run in multiple passes.
• Is it possible the headless: true isn't really "headless" for that launcher? I also tried launchOptions.args = ['-headless'] to similar effect.
• It looks like by default the launcher uses Nightly, it's possible that it's too far ahead of the build right now?

Thoughts?

[LarsDenBakker]

This is an interesting problem. You don't get it with chromium or webkit? Is this with puppeteer or playwright? For puppeteer it's still in an experimental stage, and for playwright I'm not sure how official their solution is. It would be worth filing issues for this so that is can be fixed properly.

Did this work in Karma? Of course tests all run in a single window, but if you run multiple browsers firefox would still not have the focus. @daKmoR you've had a lot of experience with focus in tests, any tips you can give here?

[daKmoR]

I think I can reproduce the behavior also via lion/form-core...
My rough guess would be that it's either a Firefox or Firefox - Playwright thing...

It works fine in chrome launcher, in playwright chromium, webkit and in puppeteer... so only firefox fails when running via "headless"

however if I watch and use debug then the tests work fine 😅

[Westbrook]

Yes, I'm experiencing this with Firefox via Playwright. This did not occur previously with Karma, even with the concurrency way up. In theory, being headless means that focus is a state of mind, but 🤷 . The launcher there used the following settings to ensure that:

            launchOptions: {
                headless: true,
                args: ['-headless'],
                firefoxUserPrefs: {
                    'toolkit.telemetry.reportingpolicy.firstRun': false,
                    'browser.shell.checkDefaultBrowser': false,
                    'browser.bookmarks.restore_default_bookmarks': false,
                    'dom.disable_open_during_load': false,
                    'dom.max_script_run_time': 0,
                    'dom.min_background_timeout_value': 10,
                    'extensions.autoDisableScopes': 0,
                    'browser.tabs.remote.autostart': false,
                    'browser.tabs.remote.autostart.2': false,
                    'extensions.enabledScopes': 15,
                },
            },

I think this is now applying these settings correctly (some of them were causing a crash previous to "@web/test-runner": "^0.6.45",) but either way they may also be a redherring.

[daKmoR]

I tested it yesterday in Firefox via browserstack as well and that runs fine 💪

Soo probably something to do how playwright starts Firefox... 🤔

what I didn't test yet is puppeteer and Firefox 🤔

[LarsDenBakker]

One possible route here is to use separate browser instances rather than separate tabs. I'm not sure what the effect will be on performance, I think we'd lose cross-page caching :(

[Westbrook]

Hrm. If it's working in browser stack and not locally then I'd think it's even more likely the prefs aren't being loaded correctly. Various parts of the prefs were needed to turn off things that were mostly happening on local Firefox builds... Particularly, things like toolkit.telemetry.reportingpolicy.firstRun and browser.shell.checkDefaultBrowser prevent the browser from throwing focus into a popup, and extensions.enabledScopes prevents an extension from loading that obfuscates the page. 🤔

[Westbrook]

But, I guess that would more likely align with it always failing locally... and right now if you reduce concurrency is doesn't have any issues. 😖

[LarsDenBakker]

Do you think you'd be able to reduce this to a minimal test case? Maybe even with puppeteer/playwright alone? That would be help experimenting with different solutions.

[Niznikr]

Wanted to mention I still notice focus related tests failing for us too on Playwright Firefox with concurrency greater than 2.

[LarsDenBakker]

If we have a small test case we can dig deeper into this.

[Niznikr]

Here's one that fails for focusing a text input component. Passes on WebKit and Chromium.

it('is able to receive focus', async () => {
    let activeElement = null;
    const onFocusIn = (event: Event): void => {
      activeElement = event.composedPath()[0];
    };
    document.addEventListener('focusin', onFocusIn);

    component['_input'].focus(); // text input in the component shadowDom
    await component.updateComplete;
    expect(activeElement === component['_input']).to.be.true;
    document.removeEventListener('focusin', onFocusIn);
  });

[LarsDenBakker]

With #710 you can set concurrency per browser launcher. For example:

import { playwrightLauncher } from '@web/test-runner-playwright';

export default /** @type {import('@web/test-runner').TestRunnerConfig} */ ({
  browsers: [
    playwrightLauncher({ product: 'chromium' }),
    playwrightLauncher({ product: 'webkit' }),
    playwrightLauncher({ product: 'firefox', concurrency: 1 }),
  ],
});

You could also use groups to only run a selection of tests with a concurrency of 1:

import { playwrightLauncher } from '@web/test-runner-playwright';

export default /** @type {import('@web/test-runner').TestRunnerConfig} */ ({
  browsers: [
    playwrightLauncher({ product: 'chromium' }),
    playwrightLauncher({ product: 'webkit' }),
    playwrightLauncher({ product: 'firefox' }),
  ],

  groups: {
    name: 'focus-tests',
    files: 'test/focus/**/*.test.js',
    browsers: [
      playwrightLauncher({ product: 'firefox', concurrency: 1 }),
    ]
  }
});