Single stress testing of parallel tests in parallel

[santigimeno]

When trying to reproduce a flaky test from the parallel folder, I've found very useful, to reproduce the error more easily, making some copies of it and then run those tests in parallel. Something along this lines:

#!/bin/bash

BASE_PATH=./test/parallel
TEST=$1
COPIES=$2

for i in `seq 1 $COPIES`;
do
  cp $BASE_PATH/$TEST.js $BASE_PATH/$TEST-$i.js
done

while true; do
  /usr/bin/python tools/test.py --mode=release parallel/${TEST}* -J
  if [ $? -ne 0 ]; then
    break
  fi
done

From what I understand the node-stress-single-test job from the CI runs the tests sequentially. Maybe trying to run them in parallel can be useful. What do you think?

[Trott]

I started to write up a lengthy pros-and-cons comment for this, but then realized that everything I was saying boiled down to a simple statement:

The two pieces of information needed on this would be "how valuable would this feature be" and "how difficult would it be to implement on Jenkins"? I don't know the answer to either of these, which makes me unqualified to have an opinion. All I know is that I've never personally needed this, but I can imagine there is value.

Anyway, mostly commenting to see if others have opinions.

[orangemocha]

While I see the value of being able to stress parallel test runs, I am not sure if this strategy would be very effective. For one, tests in the parallel folder are assumed to be safe to run in parallel with other tests, but likely not with themselves.

Does the test runner output tell us which tests are running concurrently (at the time of a failure)? If not, that's probably low hanging fruit.

[santigimeno]

Thanks for the feedback.
I understand that some tests in the parallel folder might not work well when running some of them concurrently but shouldn't they? What I'm trying to say is that lots of tests are quite similar to each other in the sense that they handle system resources: create a socket and bind it to a port, create temporal files, etc. and there are no collisions among them. Why shouldn't this work for the same test? Probably if a test can't be run in parallel with itself, it should be moved to sequential, as there can be another similar test that tries to use the same resource. I hope I'm making some kind of sense...

[orangemocha]

Why shouldn't this work for the same test?

I remember a few instances of tests that make an assumption on a resource (eg file name, pipe name) that is unique to the particular test, and I think we have considered that enough to make the test parallelizable. The goal of running tests in parallel was to speed up a test run, so we haven't had a need to run tests in parallel with themselves. If we had that need, I suppose we could make the 'parallelizable' criteria more stringent and move a few tests back to sequential.

Pinging @jbergstroem who knows more about parallel tests.

[jbergstroem]

If run in parallel tests should use their own subfolders for sockets, temporary files and similar. Not sure about common.PORT tests though. I guess it would be a good way to flesh out tests that doesn't support it?

[santigimeno]

I've read the last meeting minutes regarding this issue. I'd be really interested in doing this. How should we proceed from here? Alexis said about opening an issue in the build repository. Should I do that, or someone else? Thanks

[Trott]

@santigimeno I don't know if it matters who opens the issue. I would say you should open it and others can chime in as needed.

[orangemocha]

👍

[santigimeno]

Issue @ build group opened here: nodejs/build#331

[joaocgreis]

Current scripts in use by https://ci.nodejs.org/job/node-stress-single-test/ :

Windows

call vcbuild.bat release nosign x64
if %errorlevel% neq 0 exit /b %errorlevel%

setlocal EnableDelayedExpansion

set OK=0
set NOK=0
for /l %%i in (1,1,%RUN_TIMES%) do (
  python tools\test.py -p tap %RUN_TESTS% && set /a OK=!OK!+1 || set /a NOK=!NOK!+1
  echo %%i   OK: !OK!   NOT OK: !NOK!
)

echo on
rm -rfv test/tmp* || true
git clean -fdx || true

if %NOK% NEQ 0 (
  echo The test is flaky.
  exit /b 1
)

All others:

#!/usr/bin/env bash -ex

python ./configure
# centos5 defaults to an old python if PYTHON is not defined
# freebsd needs NPROCESSORS_ONLN without underscore
# @TODO: jbergstroem - this should use the environment variable $JOBS
# once available on all slaves
PYTHON=python make -j $(getconf _NPROCESSORS_ONLN || getconf NPROCESSORS_ONLN)
PYTHON=python make build-addons

echo "Running test ${RUN_TIMES} times" 

OK=0
NOK=0
for i in `seq $RUN_TIMES`; do
  python tools/test.py -p tap --mode=release $RUN_TESTS && OK=$(($OK+1)) || NOK=$(($NOK+1))
  echo "$i   OK: $OK   NOT OK: $NOK"
done

if [ "$NOK" != "0" ]; then
  echo The test is flaky.
  exit 1
fi