Skip to content
This repository was archived by the owner on Feb 25, 2025. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ platform_properties:
]
device_type: none
os: Windows-10

# The current android emulator config names can be found here:
# https://chromium.googlesource.com/chromium/src.git/+/HEAD/tools/android/avd/proto
# You may use those names for the android_virtual_device version.
Expand Down
8 changes: 8 additions & 0 deletions tools/engine_tool/pubspec.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -66,3 +66,11 @@ dependency_overrides:
path: ../../third_party/pkg/process_runner
smith:
path: ../../../third_party/dart/pkg/smith
source_span:
path: ../../../third_party/dart/third_party/pkg/source_span
string_scanner:
path: ../../../third_party/dart/third_party/pkg/string_scanner
term_glyph:
path: ../../../third_party/dart/third_party/pkg/term_glyph
yaml:
path: ../../../third_party/dart/third_party/pkg/yaml
221 changes: 221 additions & 0 deletions tools/pkg/engine_build_configs/lib/src/ci_yaml.dart
Original file line number Diff line number Diff line change
@@ -0,0 +1,221 @@
// Copyright 2013 The Flutter Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

import 'package:yaml/yaml.dart' as y;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For a few months, I maintained package:cocoon_scheduler which contained Dart helpers for stuff like this used by Cocoon. I ended up removing it in flutter/cocoon#1254 as Cocoon ended up being the only customer, and it was annoying to roll the versions.

For simplicity, would you be blocked from importing the Cocoon server into the engine? We could set up a script to publish new versions to pub.dev. If it pulls in too much, we could move https://github.com/flutter/cocoon/tree/main/app_dart/lib/src/model/ci_yaml into a separate package.

\cc @keyonghan

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It depends on how extensive the transitive dependencies will be, since they would all need to be explicitly brought in via the DEPS file. If it's just a few things, then that's fine, but if the cocoon scheduler has a big list of transitive deps, then that would be a lot of trouble.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@CaseyHillers I'm having trouble finding where in https://github.com/flutter/cocoon/tree/main/app_dart/lib/src/model/ci_yaml the yaml file is parsed. It looks like the code in that directory is mostly poking around in the fields of a protobuf.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It depends on how extensive the transitive dependencies will be, since they would all need to be explicitly brought in via the DEPS file. If it's just a few things, then that's fine, but if the cocoon scheduler has a big list of transitive deps, then that would be a lot of trouble.

Is there a way we can estimate that by importing it in pubspec via a git package?

@CaseyHillers I'm having trouble finding where in https://github.com/flutter/cocoon/tree/main/app_dart/lib/src/model/ci_yaml the yaml file is parsed. It looks like the code in that directory is mostly poking around in the fields of a protobuf.

ci.yaml is proto backed, so the pipeline looks like package:yaml -> toJson() -> package:protobuf mergeFromProtoJson -> ci_yaml.dart

https://github.com/flutter/cocoon/blob/main/app_dart/lib/src/service/scheduler.dart#L272

For Cocoon, it reads the file content over the network, then

    final String configContent = await githubFileContent(
      commit.slug,
      '.ci.yaml',
      ref: commit.sha!,
    );
    final YamlMap configYaml = loadYaml(configContent) as YamlMap;
    final pb.SchedulerConfig schedulerConfig = pb.SchedulerConfig()
      ..mergeFromProto3Json(configYaml)
      ..writeToBuffer();


// This file contains classes for parsing information about CI configuration
// from the .ci.yaml file at the root of the flutter/engine repository.
// The meanings of the sections and fields are documented at:
//
// https://github.com/flutter/cocoon/blob/main/CI_YAML.md
//
// The classes here don't parse every possible field, but rather only those that
// are useful for working locally in the engine repo.

const String _targetsField = 'targets';
const String _nameField = 'name';
const String _recipeField = 'recipe';
const String _propertiesField = 'properties';
const String _configNameField = 'config_name';

/// A class containing the information deserialized from the .ci.yaml file.
///
/// The file contains three sections. "enabled_branches", "platform_properties",
/// and "targets". The "enabled_branches" section is not meaningful when working
/// locally. The configurations listed in the "targets" section inherit
/// properties listed in the "platform_properties" section depending on their
/// names. The configurations listed in the "targets" section are the names,
/// recipes, build configs, etc. of the builders in CI.
class CiConfig {
/// Builds a [CiConfig] instance from parsed yaml data.
///
/// If the yaml was malformed, then `CiConfig.valid` will be false, and
/// `CiConfig.error` will be populated with an informative error message.
/// Otherwise, `CiConfig.ciTargets` will contain a mapping from target name
/// to [CiTarget] instance.
factory CiConfig.fromYaml(y.YamlNode yaml) {
if (yaml is! y.YamlMap) {
final String error = yaml.span.message('Expected a map');
return CiConfig._error(error);
}
final y.YamlMap ymap = yaml;
final y.YamlNode? targetsNode = ymap.nodes[_targetsField];
if (targetsNode == null) {
final String error = ymap.span.message('Expected a "$_targetsField" key');
return CiConfig._error(error);
}
if (targetsNode is! y.YamlList) {
final String error = targetsNode.span.message(
'Expected "$_targetsField" to be a list.',
);
return CiConfig._error(error);
}
final y.YamlList targetsList = targetsNode;

final Map<String, CiTarget> result = <String, CiTarget>{};
for (final y.YamlNode yamlTarget in targetsList.nodes) {
final CiTarget target = CiTarget.fromYaml(yamlTarget);
if (!target.valid) {
return CiConfig._error(target.error);
}
result[target.name] = target;
}

return CiConfig._(ciTargets: result);
}

CiConfig._({
required this.ciTargets,
}) : error = null;

CiConfig._error(
this.error,
) : ciTargets = <String, CiTarget>{};

/// Information about CI builder configurations, which .ci.yaml calls
/// "targets".
final Map<String, CiTarget> ciTargets;

/// An error message when this instance is invalid.
final String? error;

/// Whether this is a valid instance.
late final bool valid = error == null;
}

/// Information about the configuration of a builder on CI, which .ci.yaml
/// calls a "target".
class CiTarget {
/// Builds a [CiTarget] from parsed yaml data.
///
/// If the yaml was malformed then `CiTarget.valid` is false and
/// `CiTarget.error` contains a useful error message. Otherwise, the other
/// fields contain information about the target.
factory CiTarget.fromYaml(y.YamlNode yaml) {
if (yaml is! y.YamlMap) {
final String error = yaml.span.message('Expected a map.');
return CiTarget._error(error);
}
final y.YamlMap targetMap = yaml;
final String? name = _stringOfNode(targetMap.nodes[_nameField]);
if (name == null) {
final String error = targetMap.span.message(
'Expected map to contain a string value for key "$_nameField".',
);
return CiTarget._error(error);
}

final String? recipe = _stringOfNode(targetMap.nodes[_recipeField]);
if (recipe == null) {
final String error = targetMap.span.message(
'Expected map to contain a string value for key "$_recipeField".',
);
return CiTarget._error(error);
}

final y.YamlNode? propertiesNode = targetMap.nodes[_propertiesField];
if (propertiesNode == null) {
final String error = targetMap.span.message(
'Expected map to contain a string value for key "$_propertiesField".',
);
return CiTarget._error(error);
}
final CiTargetProperties properties = CiTargetProperties.fromYaml(
propertiesNode,
);
if (!properties.valid) {
return CiTarget._error(properties.error);
}

return CiTarget._(
name: name,
recipe: recipe,
properties: properties,
);
}

CiTarget._({
required this.name,
required this.recipe,
required this.properties,
}) : error = null;

CiTarget._error(
this.error,
) : name = '',
recipe = '',
properties = CiTargetProperties._error('Invalid');

/// The name of the builder in CI.
final String name;

/// The CI recipe used to run the build.
final String recipe;

/// The properties of the build or builder.
final CiTargetProperties properties;

/// An error message when this instance is invalid.
final String? error;

/// Whether this is a valid instance.
late final bool valid = error == null;
}

/// Various properties of a [CiTarget].
class CiTargetProperties {
/// Builds a [CiTargetProperties] instance from parsed yaml data.
///
/// If the yaml was malformed then `CiTargetProperties.valid` is false and
/// `CiTargetProperties.error` contains a useful error message. Otherwise, the
/// other fields contain information about the target properties.
factory CiTargetProperties.fromYaml(y.YamlNode yaml) {
if (yaml is! y.YamlMap) {
final String error = yaml.span.message(
'Expected "$_propertiesField" to be a map.',
);
return CiTargetProperties._error(error);
}
final y.YamlMap propertiesMap = yaml;
final String? configName = _stringOfNode(
propertiesMap.nodes[_configNameField],
);
return CiTargetProperties._(
configName: configName ?? '',
);
}

CiTargetProperties._({
required this.configName,
}) : error = null;

CiTargetProperties._error(
this.error,
) : configName = '';

/// The name of the build configuration. If the containing [CiTarget] instance
/// is using the engine_v2 recipes, then this name is the same as the name
/// of the build config json file under ci/builders.
final String configName;

/// An error message when this instance is invalid.
final String? error;

/// Whether this is a valid instance.
late final bool valid = error == null;
}

String? _stringOfNode(y.YamlNode? stringNode) {
if (stringNode == null) {
return null;
}
if (stringNode is! y.YamlScalar) {
return null;
}
final y.YamlScalar stringScalar = stringNode;
if (stringScalar.value is! String) {
return null;
}
return stringScalar.value as String;
}
10 changes: 10 additions & 0 deletions tools/pkg/engine_build_configs/pubspec.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ dependencies:
path: any
platform: any
process_runner: any
yaml: any

dev_dependencies:
async_helper: any
Expand All @@ -33,6 +34,7 @@ dev_dependencies:
process_fakes:
path: ../process_fakes
smith: any
source_span: any

dependency_overrides:
args:
Expand Down Expand Up @@ -61,3 +63,11 @@ dependency_overrides:
path: ../../../third_party/pkg/process_runner
smith:
path: ../../../../third_party/dart/pkg/smith
source_span:
path: ../../../../third_party/dart/third_party/pkg/source_span
string_scanner:
path: ../../../../third_party/dart/third_party/pkg/string_scanner
term_glyph:
path: ../../../../third_party/dart/third_party/pkg/term_glyph
yaml:
path: ../../../../third_party/dart/third_party/pkg/yaml
Loading