make compiler worker count configurable; increase default

Author: yjbanov

lib/web_ui/dev/README.md

@@ -23,13 +23,20 @@ felt build --watch
 
 If you don't want to add `felt` to your path, you can still invoke it using a relative path like `./web_ui/dev/felt <command>`
 
-## Speeding up your builds
+## Speeding up your builds and tests
+
 You can speed up your builds by using more CPU cores. Pass `-j` to specify the desired level of parallelism, like so:
+
 ```
 felt build [-w] -j 100
 ```
+
 If you are a Google employee, you can use an internal instance of Goma to parallelize your builds. Because Goma compiles code on remote servers, this option is effective even on low-powered laptops.
 
+By default, when compiling Dart code to JavaScript, we use 16 `dart2js` workers.
+If you need to increase or reduce the number of workers, set the `BUILD_MAX_WORKERS_PER_TASK`
+environment variable to the desired number.
+
 ## Running web engine tests
 To run all tests on Chrome. This will run both integration tests and the unit tests:
 

lib/web_ui/dev/test_runner.dart

@@ -356,11 +356,32 @@ class TestCommand extends Command<bool> with ArgUtils {
           '--build-filter=${path.relativeToWebUi}.browser_test.dart.js',
         ],
     ];
+    final Stopwatch stopwatch = Stopwatch()..start();
+
+    // Increase the number of worker processes.
+    //
+    // By default build_runner will limit the number of workers to 4, which
+    // doesn't take advantage of multiple CPU cores. 16 is picked because
+    // typically engine contributors will have anywhere from 4 to 56
+    // hyperthreaded CPU cores, i.e. 8-112 hardware threads, and from 16GB
+    // to 132GB of RAM, which should be both not too low and not too high.
+    //
+    // In a testing on a 32-core 132GB workstation this sped up the build
+    // from ~4min to ~1.5min.
+    String maxWorkersPerTask = io.Platform.environment.containsKey('BUILD_MAX_WORKERS_PER_TASK')
+      ? io.Platform.environment['BUILD_MAX_WORKERS_PER_TASK']
+      : '16';
+
     final int exitCode = await runProcess(
       environment.pubExecutable,
       arguments,
       workingDirectory: environment.webUiRootDir.path,
+      environment: <String, String>{
+        'BUILD_MAX_WORKERS_PER_TASK': maxWorkersPerTask,
+      },
     );
+    stopwatch.stop();
+    print('The build took ${stopwatch.elapsedMilliseconds ~/ 1000} seconds.');
 
     if (exitCode != 0) {
       io.stderr.writeln(

lib/web_ui/dev/utils.dart

@@ -41,6 +41,7 @@ Future<int> runProcess(
   List<String> arguments, {
   String workingDirectory,
   bool mustSucceed: false,
+  Map<String, String> environment = const <String, String>{},
 }) async {
   final io.Process process = await io.Process.start(
     executable,
@@ -50,6 +51,7 @@ Future<int> runProcess(
     // the process is not able to get Dart from path.
     runInShell: io.Platform.isWindows,
     mode: io.ProcessStartMode.inheritStdio,
+    environment: environment,
   );
   final int exitCode = await process.exitCode;
   if (mustSucceed && exitCode != 0) {