diff --git a/testing/scenario_app/android/README.md b/testing/scenario_app/android/README.md index a023eb3461991..c3d5e0cb7fc46 100644 --- a/testing/scenario_app/android/README.md +++ b/testing/scenario_app/android/README.md @@ -1,102 +1,365 @@ -# Scenario App: Android Tests +# Scenario App: Android Tests and Test Runner -As mentioned in the [top-level README](../README.md), this directory contains -the Android-specific native code and tests for the [scenario app](../lib). To -run the tests, you will need to build the engine with the appropriate -configuration. +End-to-end tests and test infrastructure for the Flutter engine on Android. -For example, for the latest `android` build you've made locally: +> [!IMPORTANT] +> There are several **known issues** with this test suite: +> +> - [#144407](https://github.com/flutter/flutter/issues/144407): "Newer Android" +> tests are missing expected cropping and rotation. +> - [#144365](https://github.com/flutter/flutter/issues/144365): Tests that use +> `ExternalTextureFlutterActivity` sometimes do not render. +> - [#144232](https://github.com/flutter/flutter/issues/144232): Impeller Vulkan +> sometimes hangs on emulators. +> - [#144352](https://github.com/flutter/flutter/issues/144352): Skia Gold +> sometimes is missing expected diffs. + +--- + +Top topics covered in this document include (but are not limited to): + +- [Running the Tests](#running-the-tests) +- [Contributing](#contributing) +- [Project History](#project-history) +- [Troubleshooting](#troubleshooting) +- [Getting Help](#getting-help) + +## Introduction + +This package simulates a Flutter app that uses the engine (`dart:ui`) only, in +conjunction with Android-specific embedding code that simulates the use of the +engine in a real app (such as plugins and platform views). + +A custom test runner, [`run_android_tests.dart`](bin/run_android_tests.dart), is +used to run the tests on a connected device or emulator, and report the results, +including screenshots (golden files) and logs from `adb logcat`. + +In the following architecture diagram: + +> href="https://github.com/flutter/flutter/wiki/The-Engine-architecture" +> title="The Engine architecture"> width="300" +> alt="Anatomy of a Flutter app" +> src="https://raw.githubusercontent.com/flutter/engine/main/docs/app_anatomy.svg" +> /> + +- The Dart app is represented by [`lib/main.dart`](../lib/main.dart). +- There is no framework code. +- `dart:ui` and the engine are the same as in a real app. +- Android-specific application code is in this directory + ([`android/`](./)). +- The runner is a _custom_ test runner, + [`run_android_tests.dart`](bin/run_android_tests.dart). + +### Scope of Testing + +The tests in this package are end-to-end tests that _specifically_ exercise the +engine and Android-specific embedding code. They are not [unit tests][1] for the +engine or the framework, and are not designed to test the behavior of the +framework or specific plugins, but rather to simulate the use of a framework +or plugins downstream of the engine. + +In other words, we test "does the engine work on Android?" without a dependency +on either the Flutter framework, Flutter tooling, or any specific plugins. + +[1]: ../../../shell/platform/android + +### Golden Comparisons + +Many of the Android-specific interactions with the engine are visual, such as +[external textures](https://api.flutter.dev/flutter/widgets/Texture-class.html) +or [platform views](https://docs.flutter.dev/platform-integration/android/platform-views), +and as such, the tests in this package use golden screenshot file comparisons to +verify the correctness of the engine's output. + +For example, in [`ExternalTextureTests_testMediaSurface`](https://flutter-engine-gold.skia.org/search?corpus=flutter-engine&include_ignored=false&left_filter=name%3DExternalTextureTests_testMediaSurface&max_rgba=0&min_rgba=0&negative=true¬_at_head=false&positive=true&reference_image_required=false&right_filter=&sort=descending&untriaged=true), a [video](app/src/main/assets/sample.mp4) is converted to an external texture and displayed in a Flutter app. The test takes a screenshot of the app and compares it to a golden file: + +Two pictures, the top one Flutter and the bottom Android + +_The top picture is the Flutter app, and the bottom picture is just Android._ + +See also: + +- [`tools/compare_goldens`](../../../tools/compare_goldens). + +## Prerequisites + +If you've never worked in the `flutter/engine` repository before, you will +need to setup a development environment that is quite different from a typical +Flutter app or even working on the Flutter framework. It will take roughly +30 minutes to an hour to setup an environment, depending on your familiarity +with the tools and the speed of your internet connection. + +See also: + +- [Setting up the Engine](https://github.com/flutter/flutter/wiki/Setting-up-the-Engine-development-environment) +- [Rebuilding the Tests](#rebuilding-the-tests) + +### Android SDK + +It's highly recommended to use the engine's vendored Android SDK, which once +you have the engine set up, you can find at +`$ENGINE/src/third_party/android_tools/sdk`. Testing or running with other +versions of the SDK may work, but it's _not guaranteed_, and might have +different results. + +Consider also placing this directory in the `ANDROID_HOME` environment variable: + +```sh +export ANDROID_HOME=$ENGINE/src/third_party/android_tools/sdk +``` + +### Device or Emulator + +The tests in this package require a connected device or emulator to run. The +device or emulator should be running the same version of Android as the CI +configuration (there are issues with crashes and other problems on older +emulators in particular). + +> [!CAUTION] +> +> [#144561](https://github.com/flutter/flutter/issues/144561): The emulator +> vendored in the engine checkout is old and [has a known issue with Vulkan](https://github.com/flutter/flutter/issues/144232). +> +> If you're working locally, you can update your copy by running: +> +> ```sh +> $ANDROID_HOME/cmdline-tools/latest/bin/sdkmanager --install emulator +> ``` + +### Additional Dependencies + +While not required, it is **strongly recommended** to use an IDE such as +[Android Studio](https://developer.android.com/studio) when contributing to the +Android side of the test suite, as it will provide a better development +experience such as code completion, debugging, and more: + +Screenshot of Android Studio with Autocompletion + +> [!TIP] +> +> Android Studio is expected to work _out of the box_ in this directory. +> +> If you encounter any issues, please [file a bug](#getting-help). + +## Running the Tests + +The [test runner](../bin/run_android_tests.dart) is a Dart script that installs +the test app and test suite on a connected device or emulator, runs the tests, +and reports the results including screenshots (golden files) and logs from +`adb logcat`. + +From the `$ENGINE/src/flutter` directory, run: ```sh dart ./testing/scenario_app/bin/run_android_tests.dart ``` -Or for a specific, build, such as `android_debug_unopt_arm64`: +By default when run locally, the test runner: + +- Uses the engine-wide default graphics backend for Android. +- Uses the last built Android-specific engine artifacts (i.e. `$ENGINE/src/out/android_*/`). +- Will not diff screenshots, but does save them to a logs directory. + +### Rebuilding the Tests + +If you've made changes to any file in `scenario_app`, incluing the Dart code +in `lib/` or the Android code in `android/`, you will need to rebuild the +tests before running them. + +To rebuild the `scenario_app` for the `android_debug_unopt_arm64` variant: ```sh -dart ./testing/scenario_app/bin/run_android_tests.dart --out-dir=../out/android_debug_unopt_arm64 +ninja -C out/android_debug_unopt_arm64 scenario_app ``` See also: -- [File an issue][file_issue] with the `e: scenario-app, platform-android` - labels. +- [Compiling the Engine](https://github.com/flutter/flutter/wiki/Compiling-the-engine) + +### Common Options + +A list of options can be found by running: + +```sh +dart ./testing/scenario_app/bin/run_android_tests.dart --help +``` + +Frequently used options include: + +- `--out-dir`: Which engine artifacts to use (e.g. + `--out-dir=../out/android_debug_unopt_arm64`). + +- `--logs-dir`: Where to save full `adb logcat` logs and screenshots. + +- `--[no]-enable-impeller`: Enables/disables use of the Impeller graphics + backend. -[file_issue]: https://github.com/flutter/flutter/issues/new?labels=e:%20scenario-app,engine,platform-android,fyi-android,team-engine +- `--impeller-backend`: Use a specific Impeller backend (e.g. + `--impeller-backend=opengles`). -## Debugging +- `--force-surface-producer-surface-texture`: Force the use of `SurfaceTexture`s + for plugin code that uses `SurfaceProducer`s. This instruments the same code + path that is used for Android API versions that are <= 28 without requiring + an older emulator or device. -Debugging the tests on CI is not straightforward but is being improved: +- `--smoke-test=`: Runs a specific test, instead of all tests. -- -- +### Advanced Options -Locally (or on a temporary PR for CI), you can run the tests with the -`--smoke-test` argument to run a single test by class name, which can be useful -to verify the setup: +When debugging the runner itself, you can use the `--verbose` flag to see more +detailed output, including additional options such as configuring which binary +of `adb` to use: ```sh -dart ./testing/scenario_app/bin/run_android_tests.dart --smoke-test dev.flutter.scenarios.EngineLaunchE2ETest +dart ./testing/scenario_app/bin/run_android_tests.dart --help --verbose ``` -The result of `adb logcat` and screenshots taken during the test will be stored -in a logs directory, which is either `FLUTTER_LOGS_DIR` (if set, such as on CI) -or locally in `out/.../scenario_app/logs`. +See also: -You can then view the logs and screenshots on LUCI. [For example](https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20Engine%20Drone/2003164/overview): +- [`bin/utils/options.dart`](../bin/utils/options.dart). -![Screenshot of the Logs on LUCI](https://github.com/flutter/engine/assets/168174/79dc864c-c18b-4df9-a733-fd55301cc69c) +### CI Configuration -For a full list of flags, see [the runner](../bin/README.md). +See [`ci/builders`](../../../ci/builders) and grep for `run_android_tests.dart`: -## CI Configuration +```sh +grep -r run_android_tests.dart ci/builders +``` -See [`ci/builders`](../../../ci/builders) and grep for `run_android_tests.dart`. +> [!NOTE] +> The Impeller OpenGLES backend tests are only run on staging (`bringup: true`) +> and as such are **non-blocking**. We expect to stabilize and run these tests +> as part of a wider release of the Impeller OpenGLES backend. -### Skia +#### Older Android -> [!NOTE] -> As of 2024-02-28, Flutter on Android defaults to the Skia graphics backend. +"Older Android" refers to "code paths to support older Android API levels". +Specifically, these configurations use +`--force-surface-producer-surface-texture` (see [above](#common-options) for +details). + +| Backend | CI Configuration | CI History | Skia Gold | +| ----------------- | --------------------------- | ---------------------------------------------- | ------------------------- | +| Skia | [`ci/builders`][skia-st-ci] | [Presubmit][skia-try], [Postsubmit][skia-prod] | [Skia Gold][skia-st-gold] | +| Impeller OpenGLES | [`ci/builders`][imp-st-ci] | [Staging][imp-staging] | N/A | + +[skia-try]: https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20linux_android_emulator_skia_tests +[skia-prod]: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20linux_android_emulator_skia_tests +[imp-staging]: https://ci.chromium.org/ui/p/flutter/builders/staging/Linux%20linux_android_emulator_opengles_tests +[impeller-try]: https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20linux_android_emulator_tests +[impeller-prod]: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20linux_android_emulator_tests +[skia-st-ci]: https://github.com/search?q=repo%3Aflutter%2Fengine+path%3Aci%2Fbuilders+%22--no-enable-impeller%22+%22run_android_tests.dart%22+%22--force-surface-producer-surface-texture%22&type=code +[skia-st-gold]: https://flutter-engine-gold.skia.org/search?left_filter=ForceSurfaceProducerSurfaceTexture%3Dtrue%26GraphicsBackend%3Dskia&negative=true&positive=true&right_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dskia +[imp-st-ci]: https://github.com/search?q=repo%3Aflutter%2Fengine+path%3Aci%2Fbuilders+%22--impeller-backend%3Dopengles%22+%22run_android_tests.dart%22+%22--force-surface-producer-surface-texture%22&type=code +[imp-st-gold]: https://flutter-engine-gold.skia.org/search?left_filter=ForceSurfaceProducerSurfaceTexture%3Dtrue%26GraphicsBackend%3Dimpeller-opengles&negative=true&positive=true&right_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dskia + +#### Newer Android + +| Backend | CI Configuration | CI History | Skia Gold | +| ----------------- | ---------------------------- | ------------------------------------------------------ | ------------------------ | +| Skia | [`ci/builders`][skia-ci] | [Presubmit][skia-try], [Postsubmit][skia-prod] | [Skia Gold][skia-gold] | +| Impeller OpenGLES | [`ci/builders`][imp-gles-ci] | [Staging][imp-staging] | N/A | +| Impeller Vulkan | [`ci/builders`][imp-vk-ci] | [Presubmit][impeller-try], [Postsubmit][impeller-prod] | [Skia Gold][imp-vk-gold] | + +[skia-ci]: https://github.com/search?q=repo%3Aflutter%2Fengine+path%3Aci%2Fbuilders+%22--no-enable-impeller%22+%22run_android_tests.dart%22&type=code +[skia-gold]: https://flutter-engine-gold.skia.org/search?left_filter=ForceSurfaceProducerSurfaceTexture%3Dfalse%26GraphicsBackend%3Dskia&negative=true&positive=true&right_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dskia +[imp-gles-ci]: https://github.com/search?q=repo%3Aflutter%2Fengine+path%3Aci%2Fbuilders+%22--impeller-backend%3Dopengles%22+%22run_android_tests.dart%22&type=code +[imp-vk-ci]: https://github.com/search?q=repo%3Aflutter%2Fengine+path%3Aci%2Fbuilders+%22--impeller-backend%3Dvulkan%22+%22run_android_tests.dart%22&type=code +[imp-vk-gold]: https://flutter-engine-gold.skia.org/search?left_filter=ForceSurfaceProducerSurfaceTexture%3Dfalse%26GraphicsBackend%3Dimpeller-vulkan&negative=true&positive=true&right_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dskia + +## Contributing + +[![GitHub Issues or Pull Requests by label](https://img.shields.io/github/issues/flutter/flutter/e%3A%20scenario-app)](https://github.com/flutter/flutter/issues?q=is%3Aopen+is%3Aissue+label%3A%22e%3A+scenario-app%22) + +Contributions to this package are welcome, as it is a critical part of the +engine's test suite. + +### Anatomy of a Test + +A "test" in practice is a combination of roughly 3 components: + +1. An Android _[JUnit test][]_ that configures and launches an Android activity. +2. An Android _[activity][]_, which simulates the Android side of a plugin or + platform view. +3. A Dart _[scenario][]_, which simulates the Flutter side of an application. + +[junit test]: ./app/src/androidTest/java/dev/flutter/scenariosui/DrawSolidBlueScreenTest.java +[activity]: ./app/src/main/java/dev/flutter/scenarios/ExternalTextureFlutterActivity.java +[scenario]: ../lib/src/solid_blue.dart + +While every test suite has exactly one JUnit-instrumented class, each test can +have many activities and scenarios, each with their own configuration, setup, +and assertions. Not all of this is desirable, but it is the current state of +the test suite. + +A test might also take a screenshot. See the _Skia Gold_ links in +[CI Configuration](#ci-configuration) for examples. + +### Project History + +This test suite was [originally written in 2019](https://github.com/flutter/engine/pull/10007) +with a goal of: + +> \[being\] suitable for embedders to do integration testing with - it has no +> dependencies on the flutter_tools or framework, and so will not fail/flake +> based on variances in those downstream. + +Unfortunately, the Android side of the test suite was never fully operational, +and the tests, even if failing, were accidentally be reported as passing on CI. +In 2024, as the team got closer to shipping our new graphics backend, +[Impeller](https://docs.flutter.dev/perf/impeller) on Android, it was clear that +we needed a reliable test suite for the engine on Android, particularly for +visual tests around external textures and platform views. + +So, this package was revived and updated to be a (more) reliable test suite for +the engine on Android. It's by no means complete +([contributions welcome](#contributing)), but it did successfully catch at least +one bug that would not have been detected automatically otherwise. -There are two code branches we test using `scenario_app`: +_Go forth and test the engine on Android!_ -- Older Android devices, that use `SurfaceTexture`. - - CI Configuration (TODO: Link) - - CI History (TODO: Link) - - Skia Gold (TODO: Link) -- Newer Android devices, (API 34) that use `ImageReader`. - - CI Configuration (TODO: Link) - - CI History (TODO: Link) - - Skia Gold (TODO: Link) +## Troubleshooting -### Impeller with OpenGLES +If you encounter any issues, please [file a bug](#getting-help). -There are two code branches we test using `scenario_app`: +### My test is failing on CI -- Older Android devices, that use `SurfaceTexture`. - - CI Configuration (TODO: Link) - - CI History (TODO: Link) - - Skia Gold (TODO: Link) -- Newer Android devices, (API 34) that use `ImageReader`. - - CI Configuration (TODO: Link) - - CI History (TODO: Link) - - Skia Gold (TODO: Link) +If a test is failing on CI, it's likely that the test is failing locally as +well. Try the steps in [running the Tests](#running-the-tests) to reproduce the +failure locally, and then debug the failure as you would any other test. If this +is your first time working on the engine, you may need to setup a development +environment first (see [prerequisites](#prerequisites)). -### Impeller with Vulkan +The test runner makes extensive use of logging and screenshots to help debug +failures. If you're not sure where to start, try looking at the logs and +screenshots to see if they provide any clues (you'll need them to file a bug +anyway). -There is only a single code branch we test using `scenario_app`: +`test: Android Scenario App Integration Tests (Impeller/Vulkan)` +on [LUCI](https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20Engine%20Drone/2079486/overview) produces these logs: -- Newer Android devices, (API 34) - - CI Configuration (TODO: Link) - - CI History (TODO: Link) - - Skia Gold (TODO: Link) +![Screenshot of Logs](https://github.com/flutter/flutter/assets/168174/fed8fae6-bcd2-4e6e-a074-c2c37f34a770) -## Updating Gradle dependencies +The files include a screenshot of each test, and a _full_ logcat log from the +device (you might find it easier trying the `stdout` of the test first, which +uses rudimentary log filtering). In the case of multiple runs, the logs are +prefixed with the test configuration and run attempt. -See [Updating the Embedding Dependencies](../../../tools/cipd/android_embedding_bundle/README.md). +## Getting Help -## Output validation +To suggest changes, or highlight problems, please [file an issue](https://github.com/flutter/flutter/issues/new?labels=e:%20scenario-app,engine,platform-android,fyi-android,team-engine). -The generated output will be checked against a golden file -([`expected_golden_output.txt`](./expected_golden_output.txt)) to make sure all -output was generated. A patch will be printed to stdout if they don't match. +If you're not sure where to start, or need help debugging or contributing, you +can also reach out to [`hackers-android`](https://discord.com/channels/608014603317936148/846507907876257822) on Discord, or the Flutter engine team internally. There is +no full-time maintainer of this package, so be prepared to do some of the +legwork yourself. diff --git a/testing/scenario_app/bin/README.md b/testing/scenario_app/bin/README.md index 40863acbb5a0b..9dd9b9d5b51de 100644 --- a/testing/scenario_app/bin/README.md +++ b/testing/scenario_app/bin/README.md @@ -5,59 +5,4 @@ This directory contains code specific to running Android integration tests. The tests are uploaded and run on the device using `adb`, and screenshots are captured and compared using Skia Gold (if available, for example on CI). -## Usage - -```sh -dart bin/android_integration_tests.dart -``` - -## Debugging - -When debugging, you can use the `--smoke-test` argument to run a single test -by class name, which can be useful to verify the setup. - -For example, to run the `EngineLaunchE2ETest` test: - -```sh -dart bin/android_integration_tests.dart --smoke-test dev.flutter.scenarios.EngineLaunchE2ETest -``` - -## Additional arguments - -```txt --v, --verbose Enable verbose logging --h, --help Print usage information - --[no-]enable-impeller Whether to enable Impeller as the graphics backend. If true, the - test runner will use --impeller-backend if set, otherwise the - default backend will be used. To explicitly run with the Skia - backend, set this to false (--no-enable-impeller). - --impeller-backend The graphics backend to use when --enable-impeller is true. Unlike - the similar option when launching an app, there is no fallback; - that is, either Vulkan or OpenGLES must be specified. - [vulkan (default), opengles] - --logs-dir Path to a directory where logs and screenshots are stored. - --out-dir= Path to a out/{variant} directory where the APKs are built. - Defaults to the latest updated out/ directory that starts with - "android_" if the current working directory is within the engine - repository. - --smoke-test= Fully qualified class name of a single test to run. For example try - "dev.flutter.scenarios.EngineLaunchE2ETest" or - "dev.flutter.scenariosui.ExternalTextureTests". - --output-contents-golden= Path to a file that contains the expected filenames of golden - files. If the current working directory is within the engine - repository, defaults to - ./testing/scenario_app/android/expected_golden_output.txt. -``` - -## Advanced usage - -```txt - --[no-]use-skia-gold Whether to use Skia Gold to compare screenshots. Defaults to true - on CI and false otherwise. - --adb= Path to the Android Debug Bridge (adb) executable. If the current - working directory is within the engine repository, defaults to - ./third_party/android_tools/sdk/platform-tools/adb. - --ndk-stack= Path to the NDK stack tool. Defaults to the checked-in version in - third_party/android_tools if the current working directory is - within the engine repository on a supported platform. -``` +See [running the tests](../android/README.md#running-the-tests) for more information. diff --git a/testing/scenario_app/bin/utils/options.dart b/testing/scenario_app/bin/utils/options.dart index 92eed1e136ae7..36364cf83ba39 100644 --- a/testing/scenario_app/bin/utils/options.dart +++ b/testing/scenario_app/bin/utils/options.dart @@ -163,6 +163,7 @@ extension type const Options._(ArgResults _args) { 'prefix-logs-per-run', help: 'Whether to prefix logs with a per-run unique identifier.', defaultsTo: environment.isCi, + hide: hideUnusualOptions, ) ..addOption( 'impeller-backend',