Migrate Autopublishing to GCB #5005

sealesj · 2023-09-26T17:58:17Z

Initial testing configuration for autopublishing to use Google Cloud Build.

This change requires the pub.dev publishing to migrate into a Google Cloud Build configuration file which is triggered by a flutter GCP trigger. The tagging in the release.yml will continue to execute, and the tagging push will trigger the GCB yaml.

In this state, the current autopublishing flow would still be enabled, but we could move forward with providing access to the flutter-infra GCP project to impersonate a service account for releasing

Addresses: flutter/flutter#126827

If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.

Pre-launch Checklist

If you need help, consider asking for advice on the #hackers-new channel on Discord.

flutter-dashboard · 2023-09-26T17:58:20Z

It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie or stuartmorgan on the #hackers channel in Chat (don't just cc them here, they won't see it! Use Discord!).

If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

into release-cloudbuild

sealesj · 2023-09-28T16:25:25Z

@godofredoc would flutter-infra be an appropriate project to re-use for configuring a service account to use this yaml?

godofredoc · 2023-09-28T16:53:41Z

@godofredoc would flutter-infra be an appropriate project to re-use for configuring a service account to use this yaml?

No, we need to create a new project for this.

sealesj · 2023-09-28T17:47:15Z

No, we need to create a new project for this.

Ok, the changes should be pretty straightforward on the project, ~~but I don't have the necessary permissions to make a new GCP project in the flutter parent organization~~

I'll make a new GCP project and see how far I can get with configuring the service account on that

stuartmorgan-g · 2023-10-03T18:24:25Z

cloudbuild.yml

+  # TODO @jseales - replace --dry-run on dart command with --force
+  script: | 
+    cat /secrets/temporary-pub-token.txt | dart pub token add https://pub.dev
+    dart pub publish --dry-run


Can you describe the chain of events involved for this flow, in particular where this script is run (not our CI I assume?) and how the environment on that machine is configured? What would cause this to be in the correct directory?

Thank you for those questions. I was using this starting cloudbuild configuration to test the connection with the existing github action. My thought is that the existing github action will continue executing all the steps up to the dart publish, and that the tag push will trigger the GCB yaml.

From then I am thinking in the new GCB yaml configuring a new service account with permission to publish to pub which is authenticated in the yaml would allow pub publishing.

This yaml would only need the gcloud and dart cloud builders to run on google cloud.
However, I am not sure for the directory question and need to continue testing

I've tested using a new GCP project. The connection is working, and it seems like the pub publish command in the tool workspace is working as well. The remaining step would be to now allow publishing on the pub side for each package.

The dry-run seems to be working as expected, also showing failures since these release were not tagged which is expected, and this workflow would trigger only after the tag has already been pushed by the existing release.yml workflow.

From documentation I see that invoking this tool to publish is deprecated. Would using this tool to execute pub publish be acceptable here, or would you prefer a refactoring of the publishing so that it is invoked in a different way?

Now that you have it wired up, could you walk me through the flow (i.e., my comment above)? Once we tag a plugin, what is the exact chain of events? Currently I know how everything works in our release system (e.g., we are directly controlling the version of Dart being used, our tooling is controlling the checkout, directory, etc. management), and I'd like to have an understanding (and ideally documentation in the wiki) of exactly what is happening in the new system, both so that I can reason about its correctness (it's important that the checkout have the hash corresponding to the tag, for instance; what manages that?) and so that we know what to do if something goes wrong since it will no longer be directly under repo control.

From documentation I see that invoking this tool to publish is deprecated. Would using this tool to execute pub publish be acceptable here, or would you prefer a refactoring of the publishing so that it is invoked in a different way?

In this context using pub publish directly should be fine, since our tooling will be controlling the tagging process.

Could using pub-publish-flags=--dry-run with the current publish be a solution using existing tooling?

It would work, but it's pretty wasteful since we already run publish --dry-run as part of our standard test suite, so this would pointlessly duplicate work in CI.

Why would publish without tagging be necessary?

See #5005 (comment)

@stuartmorgan Thank you for such great conversation through this issue! Could I help contribute to the necessary repo tooling changes?

From the GCP side, the configuration is good to go with the exception of security hardening after the workflow is set up. Those changes are:

requiring approval before a build executes if it is triggered manually

disable manual ability to trigger with only exception for something like members of an access on demand group

And there is also the pending task of allowing the GCP service account to be a publisher to each of the pub.dev packages.

Could I help contribute to the necessary repo tooling changes?

Sure; they are on my to-do list, but if you have cycles to get to any of it before I do that would be great, and I'm happy to review.

So to summarize, I think the repo tooling changes we need are:

a flag for publish to tag without publishing

a flag for publish to publish without tagging (maybe validating that the tag exists already, and for the current hash)

a flag to disable group-name support for --package

#6218 will add all of these.

Thank you for this update!

stuartmorgan-g · 2024-01-09T20:50:04Z

Update from triage: this is blocked on repo tooling work. I hope to spend some time on that this month.

Adds the flowing to the tool: - A new `--exact-match-only` flag to be used with `--packages` to prevent group matching (i.e., a selection like `--packages=path_provider --exact-match-only` would only run on `packages/path_provider/path_provider`, not `packages/path_provider/*`). - Two new `publish` command flags: - `--tag-for-auto-publish`, to do all the steps that `publish` currently does except for the real `pub publish`, so it would dry-run the publish and then create and push the tag if successful. - `--already-tagged`, to skip the step of adding and pushing a tag, and replace it with a check that `HEAD` already has the expected tag. This set of additions supports a workflow where the current `release` step is changed to use `--tag-for-auto-publish`, and then the separate auto-publish system would publish each package with `... publish --already-tagged --packages=<some package> --exact-match-only`. See #5005 (comment) for previous discussion/context. Part of flutter/flutter#126827

Adds the flowing to the tool: - A new `--exact-match-only` flag to be used with `--packages` to prevent group matching (i.e., a selection like `--packages=path_provider --exact-match-only` would only run on `packages/path_provider/path_provider`, not `packages/path_provider/*`). - Two new `publish` command flags: - `--tag-for-auto-publish`, to do all the steps that `publish` currently does except for the real `pub publish`, so it would dry-run the publish and then create and push the tag if successful. - `--already-tagged`, to skip the step of adding and pushing a tag, and replace it with a check that `HEAD` already has the expected tag. This set of additions supports a workflow where the current `release` step is changed to use `--tag-for-auto-publish`, and then the separate auto-publish system would publish each package with `... publish --already-tagged --packages=<some package> --exact-match-only`. See flutter#5005 (comment) for previous discussion/context. Part of flutter/flutter#126827

stuartmorgan-g · 2024-04-09T20:07:22Z

@sealesj What's the next step here now that the tool support has landed?

sealesj · 2024-04-09T20:53:20Z

@stuartmorgan Thank you for following up. I've added this to my to-do list and will be working on it soon as a P2 task

stuartmorgan-g · 2024-06-04T19:44:38Z

From triage: Is this still in development, or was it impacted by team changes?

christopherfujino · 2024-08-13T22:52:25Z

From triage: Is this still in development, or was it impacted by team changes?

TIL about this. Gonna close this as I don't plan to implement it. Feel free to re-open if you think there's value and something we should work on this year.

sealesj and others added 2 commits September 25, 2023 16:00

create cloudbuild release yml

2006790

Merge branch 'flutter:main' into release-cloudbuild

2dbc37f

sealesj added the work in progress (do not review) label Sep 26, 2023

sealesj and others added 4 commits September 26, 2023 14:00

use dry run

6350b6d

Merge branch 'release-cloudbuild' of https://github.com/sealesj/packages

87f8199

into release-cloudbuild

use dry run

df9ac86

Merge branch 'flutter:main' into release-cloudbuild

6a542d8

sealesj marked this pull request as draft September 27, 2023 18:44

sealesj removed the work in progress (do not review) label Sep 27, 2023

sealesj requested a review from stuartmorgan-g September 27, 2023 18:45

sealesj marked this pull request as ready for review September 27, 2023 20:54

sealesj requested a review from godofredoc September 28, 2023 16:24

stuartmorgan-g reviewed Oct 3, 2023

View reviewed changes

sealesj added 13 commits October 11, 2023 16:14

test build trigger

b39a715

test build trigger

c41125d

use yaml

dda86d5

test using udpated project name

1c1162a

test build trigger

2ffc296

test cloud logging only

9379efd

test build trigger

8095208

test no logging

fadc1cf

test new trigger config

935afe7

use variable test

4ebc9a8

test after added iam

7e6bcc9

test after added iam

4b3e41f

use working dir

a8e048c

sealesj added 10 commits October 12, 2023 12:26

remove accidental prefix

7bfff19

test use tool

a740f0e

dart pub get step

b9549ba

use relative path

8fb33d4

no pub get

25c579e

use project id

60ea62b

Merge branch 'release-cloudbuild-test' into release-cloudbuild

ab42ee0

comment

cfb54b2

pub get

9fe68de

parse package name from tag

753bc92

stuartmorgan-g mentioned this pull request Feb 28, 2024

[tool] Add features to support GCB auto-publish flow #6218

Merged

sealesj self-assigned this Apr 9, 2024

sealesj added the team: infra Infrastructure like devicelab, Cirrus, or LUCI. label Apr 9, 2024

christopherfujino closed this Aug 13, 2024

stuartmorgan-g mentioned this pull request Apr 16, 2025

Document why guaranteedSchedulingRepos exists flutter/flutter#167294

Closed

Migrate Autopublishing to GCB #5005

Migrate Autopublishing to GCB #5005

Uh oh!

Conversation

sealesj commented Sep 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-launch Checklist

Uh oh!

flutter-dashboard bot commented Sep 26, 2023

Uh oh!

sealesj commented Sep 28, 2023

Uh oh!

godofredoc commented Sep 28, 2023

Uh oh!

sealesj commented Sep 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sealesj Nov 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stuartmorgan-g commented Jan 9, 2024

Uh oh!

stuartmorgan-g commented Apr 9, 2024

Uh oh!

sealesj commented Apr 9, 2024

Uh oh!

stuartmorgan-g commented Jun 4, 2024

Uh oh!

christopherfujino commented Aug 13, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sealesj commented Sep 26, 2023 •

edited

Loading

sealesj commented Sep 28, 2023 •

edited

Loading

sealesj Nov 2, 2023 •

edited

Loading