Starts a .ci.yaml parser

[zanderso]

Towards flutter/flutter#132807.

This is needed so the engine tool can map the names of CI builders listed in .ci.yaml to the names of the configurations in the build config json files in ci/builders.

[zanderso]

@CaseyHillers Wondering if there's already a reasonably standalone package that I can pull in for parsing ci.yaml instead of writing this?

[CaseyHillers]

For a few months, I maintained package:cocoon_scheduler which contained Dart helpers for stuff like this used by Cocoon. I ended up removing it in flutter/cocoon#1254 as Cocoon ended up being the only customer, and it was annoying to roll the versions.

For simplicity, would you be blocked from importing the Cocoon server into the engine? We could set up a script to publish new versions to pub.dev. If it pulls in too much, we could move https://github.com/flutter/cocoon/tree/main/app_dart/lib/src/model/ci_yaml into a separate package.

\cc @keyonghan

[zanderso]

It depends on how extensive the transitive dependencies will be, since they would all need to be explicitly brought in via the DEPS file. If it's just a few things, then that's fine, but if the cocoon scheduler has a big list of transitive deps, then that would be a lot of trouble.

[zanderso]

@CaseyHillers I'm having trouble finding where in https://github.com/flutter/cocoon/tree/main/app_dart/lib/src/model/ci_yaml the yaml file is parsed. It looks like the code in that directory is mostly poking around in the fields of a protobuf.

[CaseyHillers]

It depends on how extensive the transitive dependencies will be, since they would all need to be explicitly brought in via the DEPS file. If it's just a few things, then that's fine, but if the cocoon scheduler has a big list of transitive deps, then that would be a lot of trouble.

Is there a way we can estimate that by importing it in pubspec via a git package?

@CaseyHillers I'm having trouble finding where in https://github.com/flutter/cocoon/tree/main/app_dart/lib/src/model/ci_yaml the yaml file is parsed. It looks like the code in that directory is mostly poking around in the fields of a protobuf.

ci.yaml is proto backed, so the pipeline looks like package:yaml -> toJson() -> package:protobuf mergeFromProtoJson -> ci_yaml.dart

https://github.com/flutter/cocoon/blob/main/app_dart/lib/src/service/scheduler.dart#L272

For Cocoon, it reads the file content over the network, then

    final String configContent = await githubFileContent(
      commit.slug,
      '.ci.yaml',
      ref: commit.sha!,
    );
    final YamlMap configYaml = loadYaml(configContent) as YamlMap;
    final pb.SchedulerConfig schedulerConfig = pb.SchedulerConfig()
      ..mergeFromProto3Json(configYaml)
      ..writeToBuffer();

[keyonghan]

Can we have a test to validate the logic when other unrelated configs exist? What if there exists a malformed config there? Do we want to ignore them whenever the desired ones are valid?

[zanderso]

I think if any config looks invalid, then it's okay for the whole parse to fail. The whoel input .ci.yaml has to be valid since it would fail in presubmit if it's invalid, anyway. I added another test with both a valid and invalid target to check that behavior.

[keyonghan]

LGTM