FS & zlib concurrency limit.

[damour]

Sometimes (on many concurrent sentry client calls) rss memory usage grows and don't decrease after processing.

Test script:

var Raven = require('raven');
Raven.config('https://<key>@sentry.io/<project>').install();
// Emulate big source file
var unusedData = 'AAAAAAAAAAAAAAAAAAAAAAAAAA.... 60kb here....';

var e = new Error('test message');

for (let i = 0; i < 1000; i++) {
  Raven.captureException(e);
}

setTimeout(() => {
  console.log('Memory usage', process.memoryUsage());
}, 30 * 1000);

Results:

➜ node -v
v10.10.0

➜  node test.js

1) Without limiting - 207.77mb

Memory usage { rss: 217870336,
  heapTotal: 11780096,
  heapUsed: 6615968,
  external: 104947 }

2) With limited fs & zlib (concurrency: 10) - 60.92mb

Memory usage { rss: 63889408,
  heapTotal: 10731520,
  heapUsed: 6602432,
  external: 105819 }

3) With limited fs & zlib (concurrency: 50) - 61.37mb

Memory usage { rss: 64352256,
  heapTotal: 11780096,
  heapUsed: 6605344,
  external: 105819 }

4) With limited fs & zlib (concurrency: 100) - 81.94mb

Memory usage { rss: 85929984,
  heapTotal: 10731520,
  heapUsed: 6589912,
  external: 105819 }

5) With limited fs & zlib (concurrency: 500) - 165.59mb

Memory usage { rss: 173637632,
  heapTotal: 11255808,
  heapUsed: 6574968,
  external: 104075 }

Refs:
websockets/ws#1204
https://nodejs.org/api/zlib.html#zlib_threadpool_usage

[kamilogorek]

Looks nice, thanks!

[orestisioakeimidis]

What is the status of this?

[kamilogorek]

@orestisioakeimidis we are still tracking some memory issues on Node 6 (current code is here #1795 )