AIX: v4x-staging - object factory test failures

[mhdawson]

• Version: v4.x-staging
• Platform: AIX
• Subsystem: addons

Noticed what may be intermittent failures in addons tests in v4.x-staging stream:

https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix61-ppc64/2113/console

not ok 1027 addons/02_object_factory/test
  ---
  duration_ms: 0.184
  severity: fail
  stack: |-
    module.js:327
        throw err;
        ^
    
    Error: Cannot find module './build/Release/addon'
        at Function.Module._resolveFilename (module.js:325:15)
        at Function.Module._load (module.js:276:25)
        at Module.require (module.js:353:17)
        at require (internal/module.js:12:17)
        at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test/addons/02_object_factory/test.js:4:15)
        at Module._compile (module.js:409:26)
        at Object.Module._extensions..js (module.js:416:10)
        at Module.load (module.js:343:32)
        at Function.Module._load (module.js:300:12)
        at Function.Module.runMain (module.js:441:10)
  ...

and earlier build:
https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix61-ppc64/2111/

/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test.tap:
not ok 1027 addons/02_object_factory/test
not ok 1028 addons/03_function_factory/test
not ok 1029 addons/04_wrapping_c_objects/test
not ok 1030 addons/05_factory_of_wrapped_objects/test
not ok 1031 addons/06_passing_wrapped_objects_around/test
not ok 1032 addons/07_atexit_hooks/test

not ok 1027 addons/02_object_factory/test
  ---
  duration_ms: 0.181
  severity: fail
  stack: |-
    module.js:327
        throw err;
        ^
    
    Error: Cannot find module './build/Release/addon'
        at Function.Module._resolveFilename (module.js:325:15)
        at Function.Module._load (module.js:276:25)
        at Module.require (module.js:353:17)
        at require (internal/module.js:12:17)
        at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test/addons/02_object_factory/test.js:4:15)
        at Module._compile (module.js:409:26)
        at Object.Module._extensions..js (module.js:416:10)
        at Module.load (module.js:343:32)
        at Function.Module._load (module.js:300:12)
        at Function.Module.runMain (module.js:441:10)
  ...
not ok 1028 addons/03_function_factory/test
  ---
  duration_ms: 0.186
  severity: fail
  stack: |-
    module.js:327
        throw err;
        ^
    
    Error: Cannot find module './build/Release/addon'
        at Function.Module._resolveFilename (module.js:325:15)
        at Function.Module._load (module.js:276:25)
        at Module.require (module.js:353:17)
        at require (internal/module.js:12:17)
        at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test/addons/03_function_factory/test.js:4:15)
        at Module._compile (module.js:409:26)
        at Object.Module._extensions..js (module.js:416:10)
        at Module.load (module.js:343:32)
        at Function.Module._load (module.js:300:12)
        at Function.Module.runMain (module.js:441:10)
  ...
not ok 1029 addons/04_wrapping_c_objects/test
  ---
  duration_ms: 0.173
  severity: fail
  stack: |-
    module.js:327
        throw err;
        ^
    
    Error: Cannot find module './build/Release/addon'
        at Function.Module._resolveFilename (module.js:325:15)
        at Function.Module._load (module.js:276:25)
        at Module.require (module.js:353:17)
        at require (internal/module.js:12:17)
        at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test/addons/04_wrapping_c_objects/test.js:4:15)
        at Module._compile (module.js:409:26)
        at Object.Module._extensions..js (module.js:416:10)
        at Module.load (module.js:343:32)
        at Function.Module._load (module.js:300:12)
        at Function.Module.runMain (module.js:441:10)
  ...
not ok 1030 addons/05_factory_of_wrapped_objects/test
  ---
  duration_ms: 0.189
  severity: fail
  stack: |-
    module.js:327
        throw err;
        ^
    
    Error: Cannot find module './build/Release/addon'
        at Function.Module._resolveFilename (module.js:325:15)
        at Function.Module._load (module.js:276:25)
        at Module.require (module.js:353:17)
        at require (internal/module.js:12:17)
        at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test/addons/05_factory_of_wrapped_objects/test.js:4:15)
        at Module._compile (module.js:409:26)
        at Object.Module._extensions..js (module.js:416:10)
        at Module.load (module.js:343:32)
        at Function.Module._load (module.js:300:12)
        at Function.Module.runMain (module.js:441:10)
  ...
not ok 1031 addons/06_passing_wrapped_objects_around/test
  ---
  duration_ms: 0.188
  severity: fail
  stack: |-
    module.js:327
        throw err;
        ^
    
    Error: Cannot find module './build/Release/addon'
        at Function.Module._resolveFilename (module.js:325:15)
        at Function.Module._load (module.js:276:25)
        at Module.require (module.js:353:17)
        at require (internal/module.js:12:17)
        at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test/addons/06_passing_wrapped_objects_around/test.js:4:22)
        at Module._compile (module.js:409:26)
        at Object.Module._extensions..js (module.js:416:10)
        at Module.load (module.js:343:32)
        at Function.Module._load (module.js:300:12)
        at Function.Module.runMain (module.js:441:10)
  ...
not ok 1032 addons/07_atexit_hooks/test
  ---
  duration_ms: 0.170
  severity: fail
  stack: |-
    module.js:327
        throw err;
        ^
    
    Error: Cannot find module './build/Release/addon'
        at Function.Module._resolveFilename (module.js:325:15)
        at Function.Module._load (module.js:276:25)
        at Module.require (module.js:353:17)
        at require (internal/module.js:12:17)
        at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test/addons/07_atexit_hooks/test.js:4:15)
        at Module._compile (module.js:409:26)
        at Object.Module._extensions..js (module.js:416:10)
        at Module.load (module.js:343:32)
        at Function.Module._load (module.js:300:12)
        at Function.Module.runMain (module.js:441:10)
  ...

[sxa]

I haven't been able to replicate this running the run-ci target the v4.x-staging branch on my own AIX6.1 system as yet. The logs in question do have:

/bin/sh[2]: ./node: not found.

Repeated a number of times equivalent to the number of failing tests which suggests it possible we've got an instance of the issue from #8285 - although that fix seems to be in the v4.x codebase.

[sxa]

There are two invocations of "ln -fs out/Release/node node" in the log - one of them is where I'd expect (just after the final link of the node binary) and the second is happening just before the "not found" errors are occurring. Looking to find out where that's coming from as it seems likely that the second one is what's causing the problem ...

[sxa]

The second one is occurring because the test-ci triggers the build-addons target which triggers ${NODE_EXE}

${NODE_EXE} is listed as a .PHONY target which is causing it to get triggered regardless of whether it needs to, which makes the "ln -fs out/Release/node node" get triggered again. I suspect that stopping this would resolve it. We could add a "[ ! -r node] && ln -s out/Release/node node", or remove the -f and add || true to the end of the line. Either would stop it trying to override the forced removal/recreation which I'm guessing (without being able to replicate it) is causing a problem with the way the file system operations are happening on AIX - I'm suspecting that the delete/recreate is "more atomic" on other operating systems.

(NOTE: The condition appears to still be in place on master so I'm not sure why we're only just hitting it now)