Revert "Revert "[Fuchsia] Execute most of the testing/fuchsia/test_suites.yaml on debug and release builds"" #50295

zijiehe-google-com · 2024-02-02T19:51:41Z

Following is the original change description.

This change implements a BundledTestRunner to run most of the tests in testing/fuchsia/test_suites.yaml as ExecutableTestRunner.

Tests with packages out of out/fuchsia_*_x64/ are ignored for now.
Tests with extra test command line parameters are ignored for now.

The BundledTestRunner can share most of the logic in ExecutableTestRunner and avoid reinventing the wheel.

This change also fixes the build break of fuchsia_tests in fuchsia_release_x64 which allows tests to run on the build as well.

Tests not built with AOT are filtered out with variant field in test_suites.yaml.

Bug: flutter/flutter#140179

Pre-launch Checklist

I read the Contributor Guide and followed the process outlined there for submitting PRs.
I read the Tree Hygiene wiki page, which explains my responsibilities.
I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
I listed at least one issue that this PR fixes in the description above.
I added new tests to check the change I am making or feature I am adding, or the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
I updated/added relevant documentation (doc comments with ///).
I signed the CLA.
All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

…ites.yam…" This reverts commit 824b5a7.

zijiehe-google-com · 2024-02-02T20:42:25Z

The goma cache may slowly take effect, and I cannot reproduce the slowness caused by the linux_fuchsia release builder. https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20Engine%20Drone/1933243/overview

@keyonghan Do you happen to have any suggestions of merging this change without blocking the tree?

keyonghan · 2024-02-02T20:59:21Z

The goma cache may slowly take effect, and I cannot reproduce the slowness caused by the linux_fuchsia release builder. https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20Engine%20Drone/1933243/overview

@keyonghan Do you happen to have any suggestions of merging this change without blocking the tree?

Yeah, seems some goma cache difference. The one you linked hit 100% local caches, whereas the 40min ones just hit a portion.

We can either moving the target to staging as: #50288, or increase the timeout: https://github.com/flutter/engine/blob/main/.ci.yaml#L196

Assuming this is goma cache populating related, we can try increasing the timeout temporary. At the same time, please create a bug to track to restore the timeout after validating goma cache works (targeting build time back to <30min). /cc @zanderso

zanderso · 2024-02-02T21:08:40Z

Let's not reland this on a Friday, please.

keyonghan

Temporary timeout increase LGTM.

zanderso · 2024-02-05T18:28:08Z

Is there an issue filed to track restoring the timeout to the old value?

zijiehe-google-com · 2024-02-05T18:39:04Z

Created flutter/flutter#142932 and assigned to me; also updated the TODO.

zijiehe-google-com · 2024-02-06T16:28:33Z

If you have no more comments, I will submit this change after the latest try run.

zijiehe-google-com · 2024-02-06T17:33:03Z

The last try run is still showing a reasonable build cost under 5 minutes, https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20Engine%20Drone/1947624/overview. I will go ahead and merge this change.

auto-submit · 2024-02-06T18:44:06Z

A reason for requesting a revert of flutter/engine/50295 could
not be found or the reason was not properly formatted. Begin a comment with 'Reason for revert:' to tell the bot why
this issue is being reverted.

zanderso · 2024-02-06T18:44:12Z

Reason for revert: Timing out on CI

…/test_suites.yaml on debug and release builds"" (#50295)" This reverts commit a09980e.

…a/test_suites.yaml on debug and release builds""" (#50407) Reverts "Reland "[Fuchsia] Execute most of the testing/fuchsia/test_suites.yaml on debug and release builds""" Reverts #50295 Initiated by: zanderso Reason for reverting: Timing out on CI Original PR Author: zijiehe-google-com Reviewed By: {keyonghan} This change reverts the following previous change: Original Description: Reverts #50291, flutter/flutter#142811 Following is the original change description. This change implements a BundledTestRunner to run most of the tests in testing/fuchsia/test_suites.yaml as ExecutableTestRunner. - Tests with packages out of out/fuchsia_*_x64/ are ignored for now. - Tests with extra test command line parameters are ignored for now. The BundledTestRunner can share most of the logic in ExecutableTestRunner and avoid reinventing the wheel. This change also fixes the build break of fuchsia_tests in fuchsia_release_x64 which allows tests to run on the build as well. - Tests not built with AOT are filtered out with variant field in test_suites.yaml. Bug: flutter/flutter#140179 [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style

…ia/test_suites.yaml on debug and release builds"" (flutter/engine#50295)

…143005) flutter/engine@8088863...07cdaab 2024-02-06 [email protected] Roll Dart SDK from 29265c94a6e8 to 35269fa71956 (1 revision) (flutter/engine#50402) 2024-02-06 [email protected] Add use_rbe to gclient variables for Framework Smoke Tests (flutter/engine#50403) 2024-02-06 98614782+auto-submit[bot]@users.noreply.github.com Reverts "Revert "Revert "[Fuchsia] Execute most of the testing/fuchsia/test_suites.yaml on debug and release builds""" (flutter/engine#50407) 2024-02-06 [email protected] Skip flaking test on Windows nobody is fixing. (flutter/engine#50401) 2024-02-06 98614782+auto-submit[bot]@users.noreply.github.com Reverts "[web] Fix Scene clip bounds. Trigger resize on DPR Change." (flutter/engine#50404) 2024-02-06 6844906[email protected] Revert "Revert "[Fuchsia] Execute most of the testing/fuchsia/test_suites.yaml on debug and release builds"" (flutter/engine#50295) 2024-02-06 [email protected] [Impeller] Specify if Angle or SwiftShader is being used in the title. (flutter/engine#50376) 2024-02-06 [email protected] Run all Android `scenario_app` tests, not just the smoke test. (flutter/engine#50400) 2024-02-06 [email protected] Roll Dart SDK from b62066b42af0 to 29265c94a6e8 (10 revisions) (flutter/engine#50398) 2024-02-06 [email protected] Capture `FAILURES!!!` when running Android `scenario_app` tests. (flutter/engine#50255) 2024-02-06 [email protected] [fuchsia] Bump Fuchsia's API level to 16 (flutter/engine#50358) 2024-02-06 [email protected] Roll Skia from 9e68ed6caf6d to 44106ee8edea (1 revision) (flutter/engine#50393) 2024-02-06 [email protected] Roll Skia from c29a20702356 to 9e68ed6caf6d (1 revision) (flutter/engine#50392) 2024-02-06 [email protected] [Impeller] Cache RenderPass/Framebuffer objects on the resolve texture sources. (flutter/engine#50142) 2024-02-06 [email protected] [Impeller] Do not skip the GLES render pass if the command list is empty (flutter/engine#50381) 2024-02-06 [email protected] Roll Skia from cdf214adfb4d to c29a20702356 (54 revisions) (flutter/engine#50382) 2024-02-06 [email protected] [web] Fix Scene clip bounds. Trigger resize on DPR Change. (flutter/engine#50161) 2024-02-06 [email protected] [github actions] add cherry pick workflow for engine repo (flutter/engine#50265) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll Please CC [email protected],[email protected],[email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

…visions)" (#143025) Reverts #143005 Initiated by: vashworth Reason for reverting: `Linux technical_debt` started failing on this commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20technical_debt__cost/16684/overview Original PR Author: engine-flutter-autoroll Reviewed By: {fluttergithubbot} This change reverts the following previous change: Original Description: flutter/engine@8088863...07cdaab 2024-02-06 [email protected] Roll Dart SDK from 29265c94a6e8 to 35269fa71956 (1 revision) (flutter/engine#50402) 2024-02-06 [email protected] Add use_rbe to gclient variables for Framework Smoke Tests (flutter/engine#50403) 2024-02-06 98614782+auto-submit[bot]@users.noreply.github.com Reverts "Revert "Revert "[Fuchsia] Execute most of the testing/fuchsia/test_suites.yaml on debug and release builds""" (flutter/engine#50407) 2024-02-06 [email protected] Skip flaking test on Windows nobody is fixing. (flutter/engine#50401) 2024-02-06 98614782+auto-submit[bot]@users.noreply.github.com Reverts "[web] Fix Scene clip bounds. Trigger resize on DPR Change." (flutter/engine#50404) 2024-02-06 6844906[email protected] Revert "Revert "[Fuchsia] Execute most of the testing/fuchsia/test_suites.yaml on debug and release builds"" (flutter/engine#50295) 2024-02-06 [email protected] [Impeller] Specify if Angle or SwiftShader is being used in the title. (flutter/engine#50376) 2024-02-06 [email protected] Run all Android `scenario_app` tests, not just the smoke test. (flutter/engine#50400) 2024-02-06 [email protected] Roll Dart SDK from b62066b42af0 to 29265c94a6e8 (10 revisions) (flutter/engine#50398) 2024-02-06 [email protected] Capture `FAILURES!!!` when running Android `scenario_app` tests. (flutter/engine#50255) 2024-02-06 [email protected] [fuchsia] Bump Fuchsia's API level to 16 (flutter/engine#50358) 2024-02-06 [email protected] Roll Skia from 9e68ed6caf6d to 44106ee8edea (1 revision) (flutter/engine#50393) 2024-02-06 [email protected] Roll Skia from c29a20702356 to 9e68ed6caf6d (1 revision) (flutter/engine#50392) 2024-02-06 [email protected] [Impeller] Cache RenderPass/Framebuffer objects on the resolve texture sources. (flutter/engine#50142) 2024-02-06 [email protected] [Impeller] Do not skip the GLES render pass if the command list is empty (flutter/engine#50381) 2024-02-06 [email protected] Roll Skia from cdf214adfb4d to c29a20702356 (54 revisions) (flutter/engine#50382) 2024-02-06 [email protected] [web] Fix Scene clip bounds. Trigger resize on DPR Change. (flutter/engine#50161) 2024-02-06 [email protected] [github actions] add cherry pick workflow for engine repo (flutter/engine#50265) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll Please CC [email protected],[email protected],[email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

…ia/test_suites.yaml on debug and release builds"" (flutter/engine#50295)

…sions) (#143039) Manual roll requested by [email protected] flutter/engine@8088863...1ac6beb 2024-02-07 [email protected] Roll Skia from fd55a6bd3580 to 1b266b36a4c8 (1 revision) (flutter/engine#50423) 2024-02-06 [email protected] Roll Skia from 44106ee8edea to fd55a6bd3580 (17 revisions) (flutter/engine#50420) 2024-02-06 [email protected] [Impeller] Fix pipeline attachment layout in CanRenderToTexture. (flutter/engine#50413) 2024-02-06 [email protected] Provide toStrings for Native objects (flutter/engine#50168) 2024-02-06 [email protected] Roll Dart SDK from 29265c94a6e8 to 35269fa71956 (1 revision) (flutter/engine#50402) 2024-02-06 [email protected] Add use_rbe to gclient variables for Framework Smoke Tests (flutter/engine#50403) 2024-02-06 98614782+auto-submit[bot]@users.noreply.github.com Reverts "Revert "Revert "[Fuchsia] Execute most of the testing/fuchsia/test_suites.yaml on debug and release builds""" (flutter/engine#50407) 2024-02-06 [email protected] Skip flaking test on Windows nobody is fixing. (flutter/engine#50401) 2024-02-06 98614782+auto-submit[bot]@users.noreply.github.com Reverts "[web] Fix Scene clip bounds. Trigger resize on DPR Change." (flutter/engine#50404) 2024-02-06 6844906[email protected] Revert "Revert "[Fuchsia] Execute most of the testing/fuchsia/test_suites.yaml on debug and release builds"" (flutter/engine#50295) 2024-02-06 [email protected] [Impeller] Specify if Angle or SwiftShader is being used in the title. (flutter/engine#50376) 2024-02-06 [email protected] Run all Android `scenario_app` tests, not just the smoke test. (flutter/engine#50400) 2024-02-06 [email protected] Roll Dart SDK from b62066b42af0 to 29265c94a6e8 (10 revisions) (flutter/engine#50398) 2024-02-06 [email protected] Capture `FAILURES!!!` when running Android `scenario_app` tests. (flutter/engine#50255) 2024-02-06 [email protected] [fuchsia] Bump Fuchsia's API level to 16 (flutter/engine#50358) 2024-02-06 [email protected] Roll Skia from 9e68ed6caf6d to 44106ee8edea (1 revision) (flutter/engine#50393) 2024-02-06 [email protected] Roll Skia from c29a20702356 to 9e68ed6caf6d (1 revision) (flutter/engine#50392) 2024-02-06 [email protected] [Impeller] Cache RenderPass/Framebuffer objects on the resolve texture sources. (flutter/engine#50142) 2024-02-06 [email protected] [Impeller] Do not skip the GLES render pass if the command list is empty (flutter/engine#50381) 2024-02-06 [email protected] Roll Skia from cdf214adfb4d to c29a20702356 (54 revisions) (flutter/engine#50382) 2024-02-06 [email protected] [web] Fix Scene clip bounds. Trigger resize on DPR Change. (flutter/engine#50161) 2024-02-06 [email protected] [github actions] add cherry pick workflow for engine repo (flutter/engine#50265) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll Please CC [email protected],[email protected],[email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

Revert "Revert "[Fuchsia] Execute most of the testing/fuchsia/test_su…

578ecde

…ites.yam…" This reverts commit 824b5a7.

github-actions bot added the platform-fuchsia label Feb 2, 2024

Merge branch 'main' into revert-50291-revert-50058-main

10a7307

zijiehe-google-com and others added 2 commits February 5, 2024 09:32

Merge branch 'main' into revert-50291-revert-50058-main

4e60d43

increase the timeout

4154fdf

zijiehe-google-com mentioned this pull request Feb 5, 2024

Fuchsia CIPD upload is failing frequently flutter/flutter#142811

Closed

zijiehe-google-com requested review from CaseyHillers, keyonghan and zanderso February 5, 2024 18:07

zijiehe-google-com marked this pull request as ready for review February 5, 2024 18:17

keyonghan approved these changes Feb 5, 2024

View reviewed changes

zijiehe-google-com mentioned this pull request Feb 5, 2024

Drop the timeout of fuchsia_linux from 75 minutes to 60 minutes flutter/flutter#142932

Closed

update bug id

5da2342

zijiehe-google-com added 2 commits February 5, 2024 16:06

Merge branch 'main' into revert-50291-revert-50058-main

1d39355

Merge branch 'main' into revert-50291-revert-50058-main

82a1584

zijiehe-google-com merged commit a09980e into main Feb 6, 2024

zijiehe-google-com deleted the revert-50291-revert-50058-main branch February 6, 2024 17:34

zijiehe-google-com mentioned this pull request Feb 6, 2024

Revert "[Fuchsia] Execute most of the testing/fuchsia/test_suites.yaml on debug and release builds" #50406

Closed

zanderso added the revert Label used to revert changes in a closed and merged pull request. label Feb 6, 2024

auto-submit bot removed the revert Label used to revert changes in a closed and merged pull request. label Feb 6, 2024

zanderso added the revert Label used to revert changes in a closed and merged pull request. label Feb 6, 2024

auto-submit bot pushed a commit that referenced this pull request Feb 6, 2024

Revert "Revert "Revert "[Fuchsia] Execute most of the testing/fuchsia…

56140c8

…/test_suites.yaml on debug and release builds"" (#50295)" This reverts commit a09980e.

auto-submit bot mentioned this pull request Feb 6, 2024

Reverts "Revert "Revert "[Fuchsia] Execute most of the testing/fuchsia/test_suites.yaml on debug and release builds""" #50407

Merged

8 tasks

auto-submit bot removed the revert Label used to revert changes in a closed and merged pull request. label Feb 6, 2024

engine-flutter-autoroll mentioned this pull request Feb 6, 2024

Roll Flutter Engine from 808886312e2b to 07cdaab7f531 (18 revisions) flutter/flutter#143005

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Feb 6, 2024

a09980eec Revert "Revert "[Fuchsia] Execute most of the testing/fuchs…

2d3b282

…ia/test_suites.yaml on debug and release builds"" (flutter/engine#50295)

auto-submit bot mentioned this pull request Feb 6, 2024

Reverts "Roll Flutter Engine from 808886312e2b to 07cdaab7f531 (18 revisions)" flutter/flutter#143025

Merged

engine-flutter-autoroll mentioned this pull request Feb 7, 2024

Roll Flutter Engine from 808886312e2b to 9c67120fa023 (21 revisions) flutter/flutter#143029

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Feb 7, 2024

a09980eec Revert "Revert "[Fuchsia] Execute most of the testing/fuchs…

af29224

…ia/test_suites.yaml on debug and release builds"" (flutter/engine#50295)

engine-flutter-autoroll mentioned this pull request Feb 7, 2024

Manual roll Flutter Engine from 808886312e2b to 1ac6beb8a3c2 (22 revisions) flutter/flutter#143039

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Feb 7, 2024

a09980eec Revert "Revert "[Fuchsia] Execute most of the testing/fuchs…

7fe275a

…ia/test_suites.yaml on debug and release builds"" (flutter/engine#50295)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Revert "Revert "[Fuchsia] Execute most of the testing/fuchsia/test_suites.yaml on debug and release builds"" #50295

Revert "Revert "[Fuchsia] Execute most of the testing/fuchsia/test_suites.yaml on debug and release builds"" #50295

Uh oh!

zijiehe-google-com commented Feb 2, 2024 •

edited

Loading

Uh oh!

zijiehe-google-com commented Feb 2, 2024

Uh oh!

keyonghan commented Feb 2, 2024 •

edited

Loading

Uh oh!

zanderso commented Feb 2, 2024

Uh oh!

keyonghan left a comment

Uh oh!

zanderso commented Feb 5, 2024

Uh oh!

zijiehe-google-com commented Feb 5, 2024 •

edited

Loading

Uh oh!

zijiehe-google-com commented Feb 6, 2024

Uh oh!

zijiehe-google-com commented Feb 6, 2024

Uh oh!

auto-submit bot commented Feb 6, 2024

Uh oh!

zanderso commented Feb 6, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Revert "Revert "[Fuchsia] Execute most of the testing/fuchsia/test_suites.yaml on debug and release builds"" #50295

Revert "Revert "[Fuchsia] Execute most of the testing/fuchsia/test_suites.yaml on debug and release builds"" #50295

Uh oh!

Conversation

zijiehe-google-com commented Feb 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-launch Checklist

Uh oh!

zijiehe-google-com commented Feb 2, 2024

Uh oh!

keyonghan commented Feb 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zanderso commented Feb 2, 2024

Uh oh!

keyonghan left a comment

Choose a reason for hiding this comment

Uh oh!

zanderso commented Feb 5, 2024

Uh oh!

zijiehe-google-com commented Feb 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zijiehe-google-com commented Feb 6, 2024

Uh oh!

zijiehe-google-com commented Feb 6, 2024

Uh oh!

auto-submit bot commented Feb 6, 2024

Uh oh!

zanderso commented Feb 6, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zijiehe-google-com commented Feb 2, 2024 •

edited

Loading

keyonghan commented Feb 2, 2024 •

edited

Loading

zijiehe-google-com commented Feb 5, 2024 •

edited

Loading