New CI job to stress a single test

[joaocgreis]

@nodejs/collaborators

https://ci.nodejs.org/job/node-stress-single-test/ is now available and ready to use. It can be used to help determining if a test is flaky or not.

To test a pull request, use refs/pull/99999/head in the GIT_REMOTE_REF field.

The default of 100 runs can be useful to get the error output of a test that fails frequently, or to quickly test a fix attempt before a longer test. To know with a high probability that a test is not flaky I suggest 10000 runs, because I've seen a flaky test failing about 3 out of 10000 runs. For now these are just suggestions, we can change the default and create best practices when we have more experience with this.

[jasnell]

+1! Awesome to see. Thank you!
On Nov 16, 2015 5:25 AM, "João Reis" notifications@github.com wrote:

@nodejs/collaborators https://github.com/orgs/nodejs/teams/collaborators

https://ci.nodejs.org/job/node-stress-single-test/ is now available and
ready to use. It can be used to help determining if a test is flaky or not.

To test a pull request, use refs/pull/99999/head in the GIT_REMOTE_REF
field.

The default of 100 runs can be useful to get the error output of a test
that fails frequently, or to quickly test a fix attempt before a longer
test. To know with a high probability that a test is not flaky I suggest
10000 runs, because I've seen a flaky test failing about 3 out of 10000
runs. For now these are just suggestions, we can change the default and
create best practices when we have more experience with this.

—
Reply to this email directly or view it on GitHub
#3854.

[Fishrock123]

Wow, nice job @joaocgreis!

[cjihrig]

This is cool! Please don't take me closing the issue as thinking otherwise :-)

[Trott]

This is great, and I'm using this RIGHT NOW to determine one test's flaky-or-not status. Thanks!

Super-minor nit: 9999 is easier to read and type than 10000, so maybe recommend that in the tiny text beneath the box for RUN_TIMES?

[joaocgreis]

Thanks!

@Trott fixed!

[jasnell]

@joaocgreis .. how difficult do you think it would be to create a variation on this that is essentially a combination of node-test-pull-request and node-stress-single-test. For Pull-Requests that make test changes, it would be nice to be able to do a node-test-pull-request and stress test all at once (see: #3758 (comment) and subsequent comments)

[joaocgreis]

@jasnell that would involve changing all the test-commit jobs. A lot of jenkins clicking, but quite doable. The problem I see would be to include it in the TAP output. The best thing would be to stress the test with test.py, running it along with other tests as I described in nodejs/build#260 (comment) .

[jasnell]

OK, good to know. I'll add it to my list of things to investigate when I
get a bit more time
On Nov 18, 2015 10:12 AM, "João Reis" notifications@github.com wrote:

@jasnell https://github.com/jasnell that would involve changing all the
test-commit jobs. A lot of jenkins clicking, but quite doable. The problem
I see would be to include it in the TAP output. The best thing would be to
stress the test with test.py, running it along with other tests as I
described in nodejs/build#260 (comment)
nodejs/build#260 (comment) .

—
Reply to this email directly or view it on GitHub
#3854 (comment).