extremely slow build

[shreyas1599]

Running flutter pub run build_runner build --delete-conflicting-outputs at least 15-20 minutes for every change in a graphql file. I tried modifying build.yaml and adding generate_for but it does not seem to make any difference.

ferry_generator|req_builder:
        generate_for:
          - lib/graphql/*.* 

I assumed using generate_for will skip running on files not within the specified directory but I still get the following:

[WARNING] No actions completed for 42.2s, waiting on: <files_not_in_generate_for>

Is there any field in options that I can use to run build_runner only for a specific directory or any recommendations to speed up the build? Here's the entire build.yaml.

Thanks a lot!

[smkhalsa]

Running flutter pub run build_runner build --delete-conflicting-outputs at least 15-20 minutes for every change in a graphql file.

Wow. I thought my build was taking too long at ~3 minutes.

I believe there are two factors contributing to the slow build times:

1. built_value generates a lot of boilerplate (most of which is generated by the built_value_generator)
2. build_runner isn't great about caching / selectively rebuilding only what has changed

With respect to "1", if and when Dart gets records, we'd likely be able to eliminate built_value altogether. However, that probably won't happen very soon.

"2" is really beyond the scope of ferry. There may already be some solutions that ferry users can use to improve this, but I haven't spent much time investigating it. If you find that generate_for or any other configuration enables you to reduce your build time, we should definitely add a note to the docs.

[shreyas1599]

Ah, I see. Thanks for your reply. Another problem with the 15 minutes is that my computer hangs and if I try to do anything else while it is running (one tab open in the browser) the build takes a few minutes more. Maybe my computer is slow(but I've found it bearable with two emulators open and running and it is unusable while I run the build :/).

generate_for doesn't seem to be working. I'll try and see if I can find something to speed up the process and comment here. 15 minutes is just too long to get anything done :/

(Sidenote: ferry is great! Thanks for this!)

[knaeckeKami]

@shreyas1599 If it helps you, I moved my graphql api in a separate dart project.
That way I only have to the re-run build_runner for ferry when the actual grapql-related code changes, which is not that often (at least for me).
Also, it prevents dev-dependency conflicts with other code generation packages.

I also use this script to put all the generated graphql code in a single library (but look out for naming clashes):

import 'dart:io';


void main() {
  final suffix = RegExp(r'^.+(req|var|data)\.gql\.dart$');

  if(Directory.current.path.endsWith('bin')){
    Directory.current = Directory.current.parent;
  }

  final files = Directory('lib')
      .listSync(recursive: true)
      .whereType<File>()
      .where((element) => suffix.hasMatch(element.path));

  print('');
  // export the schema file 
  print("export 'src/schema/schema.schema.gql.dart';");

  for(final file in files){
    print("export '${file.path.substring(4)}';");
  }
}

[friebetill]

I was able to reduce my build time from 15 minutes back to 3 minutes by measuring and then reducing the memory consumption in VSCode.

[shreyas1599]

With some of the suggestions mentioned here, I was able to get it down to under 7 minutes. Seems manageable. I don't think I can get it down further as my computer seems to have slowed down due to age. Closing this. Thanks for the help.

[TMInnovations]

Hi everyone,

I also use ferry for one project with a 11.000 lines schema (still growing). The last generation took about 10 hours with almost 100% RAM and 2 of 8 kernels 100% used. Finally the build broke with a message like "out of memory".

I also tried to run the generation process inside a github runner - same result.

VSCode RAM reducing techniques were applied.

Is there a way to change the generator source code to cope with growing schema files? Ferry is really nice but it wants my RAM and CPU to scale with the schema file size, which is quite a limitation.

If there are ideas on how to solve this problem please let me know. If you could tell me where to start, I'd also be happy to contribute to the ferry project.

All the best,
Thomas

Edit 1:
Schema on its own takes about 30+ minutes. With about 25 queries and 15 mutations it takes about 4 hours.
Versions are:

• Flutter: 2.10.4
• ferry_generator: ^0.5.0-dev.10
• gql_code_builder: ^0.5.1-alpha+1645425888395
• build_runner: ^2.1.7

Generation always stops with logs trying to blame "built_value_generator:built_value"

Any idea on how we can save RAM in this build process?

[michael-golfi]

I have been seeing similar long codegen lengths and posted this issue, and a few things I tried to resolve it.

Edit:

For anyone looking for steps to repro my performance measurement approach. Try the running the following script:

export BUILD_MAX_WORKERS_PER_TASK=1

echo "Clearing old build_runner state"
flutter clean &> /dev/null

echo "Removing old generated files"
rm -rf lib/__generated__ \
       lib/src/graphql/resolvers/__generated__ \
       lib/src/graphql/resolvers/fragments/__generated__

echo "Installing packages"
# Note you need these packages in your dev_dependencies to run build_runner with `--track-performance`
# build_test: any
# build_web_compilers: any
flutter pub get

echo "Running build runner with $BUILD_MAX_WORKERS_PER_TASK parallel workers"
flutter pub run build_runner serve --delete-conflicting-outputs --track-performance

# Open a browser window to http://localhost:8080/$perf
# Note the page will only actually load once build_runner completes

[knaeckeKami]

there was just a new release of built_value that might help somewhat.

if this is still an issue for you, could you try

built_value_generator: ^8.4.3

and see if this improves things?

[dehypnosis]

It will be a significant bottleneck for this awesome project.
Do we have any plan or alternative?

[knaeckeKami]

Unfortunately, to me, it seems the bottleneck is either built_value or build_runner with big files in general.

Potentially it could be feasible to move away from built_value and let ferry generate all code, but this would be a big task which I do not have the capacities for right now.

[dehypnosis]

@knaeckeKami Thank you for your answer.

But what I think is that the code generation strategy has a point to be improved.
I want to share you a point found while I am digging this issue.

In GraphQL, using a Fragment repeatedly is common. And some fragments can include other fragments too.
Currently gql_code_builder generates all the fields with selectionSet as a separated built value class.
It makes graph-way approach inefficient and requires too many code generation without reusing models of exactly same logics.

for exampe..

fragment FooFragment on Foo {
   a
   b
   c
}

fragment BarFragment on Bar {
   a
   b
   c
   foo {
      ...FooFragment
   }
}

In this case, any model with BarFragment will generates duplicate foo { ...FooFragment }.
With more complexity, amount of code generation increases exponentially.

for example, my case is..

scheme $ wc -l *
      51 query.admin.graphql
     258 query.community.graphql
     565 query.graphql
      67 scalar.dart
      11 schema.dart
    1095 schema.graphql
    2047 total

generated $ wc -l *
     364 query.admin.ast.gql.dart
    1810 query.admin.data.gql.dart
   11845 query.admin.data.gql.g.dart
     292 query.admin.req.gql.dart
    1663 query.admin.req.gql.g.dart
     122 query.admin.var.gql.dart
     643 query.admin.var.gql.g.dart
    3520 query.ast.gql.dart
    1661 query.community.ast.gql.dart
   58155 query.community.data.gql.dart
  348729 query.community.data.gql.g.dart
    1216 query.community.req.gql.dart
    7084 query.community.req.gql.g.dart
     563 query.community.var.gql.dart
    3129 query.community.var.gql.g.dart
   18976 query.data.gql.dart
  112482 query.data.gql.g.dart
    2654 query.req.gql.dart
   15563 query.req.gql.g.dart
    1204 query.var.gql.dart
    6348 query.var.gql.g.dart
    8735 schema.ast.gql.dart
    1613 schema.schema.gql.dart
    9783 schema.schema.gql.g.dart
    2829 serializers.gql.dart
    6097 serializers.gql.g.dart
  627080 total

about 2000 line of gql code generates 627080 line of dart codes..
even though some of my query is quite complex, it is hard to say that it efficiently generate codes.

I am working on refactoring part ofgql_code_builder: https://github.com/gql-dart/gql/blob/e021a3774f77701a534c427d702df4b7781c4a66/codegen/gql_code_builder/lib/src/operation/data.dart#L120
to reduce redundant code generations.

FYI, I just modified code to not to generate a data class for field with a single FragmentSpreadNode selectionSet. And make that field reuse already generated ..FragmentData class. And the result is..

generated $ wc -l *
     364 query.admin.ast.gql.dart
     982 query.admin.data.gql.dart
     292 query.admin.req.gql.dart
    1663 query.admin.req.gql.g.dart
     122 query.admin.var.gql.dart
     643 query.admin.var.gql.g.dart
    3520 query.ast.gql.dart
    1661 query.community.ast.gql.dart
    2517 query.community.data.gql.dart
    1216 query.community.req.gql.dart
    7084 query.community.req.gql.g.dart
     563 query.community.var.gql.dart
    3129 query.community.var.gql.g.dart
    4569 query.data.gql.dart
   29187 query.data.gql.g.dart
    2654 query.req.gql.dart
   15563 query.req.gql.g.dart
    1204 query.var.gql.dart
    6348 query.var.gql.g.dart
    8735 schema.ast.gql.dart
    1613 schema.schema.gql.dart
    9783 schema.schema.gql.g.dart
     787 serializers.gql.dart
     927 serializers.gql.g.dart
  105126 total

It reduced code generation to 16% of original one.
And originally a build took about 15-20min, but this build takes me only 1-2min.

However this is not finished yet and I don't have much background of GraphQL AST and gql-dart/built-value/ferry specifications to make it complete.
Could you consider such approach together? I think It might not take a lot of time.