Support for node.js workers

[dschuff]

Work has been started on workers in node.js (nodejs/node#20876). The idea looks great, and we should make sure our pthreads code works there too. The API (which presumably isn't final) looks more or less like web workers, so it shouldn't be too hard to make it work.

[dschuff]

I had forgotten that someone had already filed a bug about this previously: #6332 In any case, we should make this work for node.js as well as browsers.

[kripken]

Experimenting with this, it is blocked on nodejs/node#25630 - looks like in node a blocking main does not send postMessages to workers (or perhaps blocks all logging from them).

[bnoordhuis]

or perhaps blocks all logging from them

Just to confirm: that's what happens.

You can postMessage from main to worker, no problem, but console.log() in a worker sends the string to the main thread, to print it on the worker's behalf.

The main thread won't process that string until control returns to the event loop and fixing that is not trivial.

[kripken]

I see, thanks @bnoordhuis!

Maybe we can work around it by using stdout/stderr directly? Or would that not work either?

[bnoordhuis]

No, unfortunately. All stdio is mediated in Node's workers.

[kripken]

I see, thanks. Ok, sadly I think that means we can't use node workers in a simple way here.

[walkingeyerobot]

I believe bleeding edge Node should have all the support necessary.

[kripken]

Latest node may be enough, yeah. Last I tried it seemed possible, but getting logging working from a pthread (while the main is busy) was hard, so it was difficult to make progress. I have a wip branch here:

https://github.com/emscripten-core/emscripten/tree/node-pthreads

I may not have time for this myself in the near future. If someone has time to look at that that would be great, it might be a matter of finding some hack for debugging (like maybe printing to a file?), then fixing some minor issues.

[kripken]

I looked at this a little more now, there is some progress in the branch. Overall I keep hitting the many small differences between a Web Worker and a Node Worker.

For example the current blocker is that on the web we use importScripts to load the main x.js from the x.worker.js. Node doesn't have importScripts. The closest I can find is to load the code and do a global eval, but when doing that, require() is not defined, and we need that to be able to read files etc. Any ideas?

[addaleax]

@kripken Maybe I’m misunderstanding, but what’s a “global eval” in that context? eval('require') should give the require function as long as the outer context already has it.

[kripken]

@addaleax

It's something like eval.call(null, '...'), which can add stuff to the global scope (and not just the current function scope if there is one), which we need to match importScripts. The emscripten code is here and here's an example of why it's needed:

function setup() {
  eval('function foo() {}'); // normal eval
  console.log(typeof foo); // ok
  eval.call(null, 'function baz() {}'); // global eval
  console.log(typeof baz); // ok
}
setup();
console.log(typeof foo); // bad!
console.log(typeof baz); // ok

Turns out in node a normal eval does see require, but a global one doesn't :( But it does work in V8 and SpiderMonkey...

Maybe there's a better way I'm missing?

[addaleax]

Okay, I see – thanks for the explanation.

In that case, maybe doing something like we do for the REPL in Node.js is a good approach? Basically, we copy/provide Node’s special per-module variables, require, module, exports, require, __dirname and __filename to the global object, and then run eval() on the script.

I think that’s something that makes sense when your goal is to have everything in the global scope, as it would be in a Web Worker?

[kripken]

Interesting, thanks - so there's a way I can copy require to the global scope, and global eval would see it? How would I do that copy?