From 025978832dc9a56e552ef0b02fbac010fcf58c56 Mon Sep 17 00:00:00 2001 From: Dean Ellis Date: Mon, 27 Feb 2023 10:07:25 +0000 Subject: [PATCH 01/12] Add UnitTest docs --- Documentation/workflow/UnitTests.md | 184 ++++++++++++++++++++++++++++ 1 file changed, 184 insertions(+) create mode 100644 Documentation/workflow/UnitTests.md diff --git a/Documentation/workflow/UnitTests.md b/Documentation/workflow/UnitTests.md new file mode 100644 index 00000000000..b86cf693541 --- /dev/null +++ b/Documentation/workflow/UnitTests.md @@ -0,0 +1,184 @@ +# Unit Tests + + +## Tools + +1. [NUnit](https://github.com/nunit/nunit) +2. [XUnit](https://github.com/xunit/xunit) + +## Project Test Types + +1. Build Tests + + There are a number of `Build` related tests which test various aspects of the `.NET Android` build system. This includes not only MSBuild Task tests but build integration tests. Since the SDK is consumed from MSBuild, almost all of the unit tests are MSBuild related. + +2. Device Tests + + Because `.NET Android` has to work on devices, we have a number of + "Integration" tests which check to make sure a final app will function as expected on an Emulator or Device. Our CI system will + test against an Emulator. However the system will pick up the first attached device, so developers can test one physical hardware + if needed. + +## Test Count + +| Test | Count | +| :---------: | ----: | +| Test | 0 | +| Test | 100 | + + +## Running Tests + +Running tests in the IDE is currently not supported. + +After building the repo you can make use of the `dotnet-local` script +to run unit tests against the locally build SDK. The `dotnet-local` script is a wrapper around the custom `dotnet` installation which the +build downloads and installs in `bin/Debug/dotnet` or `bin/Release/dotnet` (depending on your configuration). + +### MacOS/Linux + +To run ALL the build tests run. + +`dotnet-local.sh test bin/TestDebug/net7.0/Xamarin.Android.Build.Tests.dll --filter=Category!=DotNetIgnore` + +To run ALL the supported Device Integraton tests runs. + +`dotnet-local.sh test bin/TestDebug/MSBuildDeviceIntegration/net7.0/MSBuildDeviceIntegration.dll --filter=Category!=DotNetIgnore` + +NOTE: Not all tests work under .NET Android yet. So we need to filter +them on the `DotNetIgnore` category. + + +### Windows + +On Windows we can use the same `dotnet-local` script to run the tests +just like we do on other platforms. + +`dotnet-local.cmd test bin/TestDebug/net7.0/Xamarin.Android.Build.Tests.dll --filter=Category!=DotNetIgnore` + +`dotnet-local.cmd test bin/TestDebug/MSBuildDeviceIntegration/net7.0/MSBuildDeviceIntegration.dll --filter=Category!=DotNetIgnore` + +## Writing Tests + +This section outlines how to write the unit tests for the various parts of the SDK. +Any new test `class` should derive from `BaseTest` or in the case of Device based tests, `DeviceTest`. These base classes provide additional helper methods to +create and run the unit tests. They also projects methods to run things like +`adb` commands and to auto cleanup the unit tests. They will also capture additional +things like screenshots if a test fails. + +### Task Tests + +Tests which run on MSBuild Tasks are located in the `src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Xamarin.Android.Build.Tests.csproj` project. They should be placed in the `src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Tasks/` folder along with the other tests. + +There is an implementation of the `IBuildEngine*` interfaces in the `MockBuildEngine` class. You can use this to mock the MSBuild +runtime and test tasks directly. You might need to create an instance of the `MockBuildEngine` per test since it captures warnings and errors to specific collections provided to the constructor. So if you are testing if a `Task` produces a specific error it will need its own `MockBuildEngine`. Just in case the test is run in parallel. + +```csharp +var engine = new MockBuildEngine (TestContext.Out); +``` + +Once you have a `MockBuildEngine` you can then create an instance +of your `Task` and then assign the `BuildEngine` property. + +```csharp +var task = new MyTask () { + BuildEngine = engine, +}; +``` + +Then you can `Assert` on the `Execute` method of the task. This will run the task and return a `bool`. + +```csharp +Assert.IsTrue (task.Execute (), "Task should succeed."); +``` + +NOTE: It is common practice in .NET Android to provide a text description on an `Assert`. This makes it easier to track down +where a particular test is failing. + +If you want to capture warnings and errors you need to provide the `MockBuildEngine` with the appropriate arguments. + +```csharp +var errors = new List; +var warnings = new List; +var messages = new List; +var engine = new MockBuildEngine (TestContext.Out, errors: errors, warnings: warnings messages:messages); +``` + +You can then check these collections for specific output from the `Task`. + +Putting it all together + +```csharp + +[Test] +public void MyTaskShouldSucceedWithNoWarnings +{ + var warnings = new List; + var messages = new List; + var engine = new MockBuildEngine (TestContext.Out, warnings: warnings); + var task = new MyTask () { + BuildEngine = engine, + }; + Assert.IsTrue (task.Execute (), "Task should succeed."); + Assert.AreEqual (0, warnings.Count, $"Task should not emit any warnings, found {warnings.Count}"); +} +``` + +### Build Tests + +Tests which need to test the SDK integration are located in the `src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Xamarin.Android.Build.Tests.csproj` project. These types of test do NOT run on a Device. Device tests are slow and expensive to run (time wise). Generally these tests check that apps can build and produce the correct files in the final `apk`. It is also where we add tests for specific user reported issues, for example build errors around non ASCII characters etc. + +There are other build tests which test other aspects of the SDK. Examples are +`tests/CodeBehind/BuildTests/CodeBehindBuildTests.csproj` + +Writing a test makes use of the `src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Xamarin.ProjectTools.csproj` API. This api exposes a way to programmatically generating csproj files as well as other application based source code. This saves us from having to have 1000's of csproj files all over the repo. + +At its core you create an `XamarinAndroidProject` instance. This can be +`XamarinAndroidApplicationProject` or say `XamarinFormsApplicationProject`. + +```csharp +var project = new XamarinAndroidApplicationProject (); +``` + +You can then Add Items such as source files, images or other files. By default it +will create a simple Android App which will include one `MainActivity.cs` and some +standard resources. Properties can be set via the `SetProperty` method. This can be +done globally or for a specific Configuration. + +```csharp +project.SetProperty ("MyGlobalBoolProperty", "False"); +project.SetProperty (project.DebugConfiguration, "MyDebugBoolProperty", "False"); +``` + +Once you have a project object constructed, you can make use of the `ProjectBuilder` +to build the project. There are two helper methods `CreateApkBuilder` and `CreateDllBuilder` which are available. These will allow you do create a builder to +output an `apk` or in the base of a `Library` project a `dll`. +You call `CreateApkBuilder` to create the builder then pass the project to the `Build` +method. This will build the project. There are other methods such as `Save` and `Install` which can be used to run the various underlying MSBuild targets. +NOTE: You should wrap your instances of a `ProjectBuilder` inside a `using` to make sure that the files are cleaned up correctly after the test has run. Tests which fail +will leave their files on disk to later inspection or archiving. + +```csharp +using (var builder = new CreateApkBuilder ()) { + Assert.IsTrue (builder.Build (proj), "Build should have succeeded."); +} +``` + +When running under .NET the ProjectTools will automatically switch to using SDK style +projects and will generate .NET based projects. When running under `msbuild` it will +generate the old style projects. This allows you do write the same test for both types +of SDK. + + + + +### Device Tests + +Device based tests are located in `tests/MSBuildDeviceIntegration/MSBuildDeviceIntegration.csproj`. These work in a similar fashion to the other MSBuild +related tests. The only requirement is that they need a Device Attached. + +### On Device Unit Tests + +There are a category of tests which run on the device itself, these tests the runtime +behaviour. These run `NUnit` and `XUnit` tests directly on the device. Some of these are located in the runtime itself. We build them within the repo then run the tests on +the device. From ac0ed6429ecc3450a3849ff06ef8ae0578b55036 Mon Sep 17 00:00:00 2001 From: Dean Ellis Date: Mon, 13 Mar 2023 11:55:30 +0000 Subject: [PATCH 02/12] Updated docs --- Documentation/workflow/UnitTests.md | 66 +++++++++++++++++++++++++---- 1 file changed, 57 insertions(+), 9 deletions(-) diff --git a/Documentation/workflow/UnitTests.md b/Documentation/workflow/UnitTests.md index b86cf693541..75a6b697c05 100644 --- a/Documentation/workflow/UnitTests.md +++ b/Documentation/workflow/UnitTests.md @@ -62,7 +62,7 @@ just like we do on other platforms. This section outlines how to write the unit tests for the various parts of the SDK. Any new test `class` should derive from `BaseTest` or in the case of Device based tests, `DeviceTest`. These base classes provide additional helper methods to -create and run the unit tests. They also projects methods to run things like +create and run the unit tests. They also contain methods to run things like `adb` commands and to auto cleanup the unit tests. They will also capture additional things like screenshots if a test fails. @@ -109,7 +109,24 @@ You can then check these collections for specific output from the `Task`. Putting it all together ```csharp +[Test] +public void MyTaskShouldSucceedWithNoWarnings +{ + var warnings = new List; + var messages = new List; + var engine = new MockBuildEngine (TestContext.Out, warnings: warnings); + var task = new MyTask () { + BuildEngine = engine, + }; + Assert.IsTrue (task.Execute (), "Task should succeed."); + Assert.AreEqual (0, warnings.Count, $"Task should not emit any warnings, found {warnings.Count}"); +} +``` +Adding `ITaskItem` properties to the Task can be done just like setting normal properties. This way you +can test out all sorts of scenarios. + +```csharp [Test] public void MyTaskShouldSucceedWithNoWarnings { @@ -118,6 +135,7 @@ public void MyTaskShouldSucceedWithNoWarnings var engine = new MockBuildEngine (TestContext.Out, warnings: warnings); var task = new MyTask () { BuildEngine = engine, + SomeItem = new TaskItem ("somefile.txt"), }; Assert.IsTrue (task.Execute (), "Task should succeed."); Assert.AreEqual (0, warnings.Count, $"Task should not emit any warnings, found {warnings.Count}"); @@ -131,7 +149,7 @@ Tests which need to test the SDK integration are located in the `src/Xamarin.And There are other build tests which test other aspects of the SDK. Examples are `tests/CodeBehind/BuildTests/CodeBehindBuildTests.csproj` -Writing a test makes use of the `src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Xamarin.ProjectTools.csproj` API. This api exposes a way to programmatically generating csproj files as well as other application based source code. This saves us from having to have 1000's of csproj files all over the repo. +Writing a test makes use of the `src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Xamarin.ProjectTools.csproj` API. This API exposes a way to programmatically generating csproj files as well as other application based source code. This saves us from having to have 1000's of csproj files all over the repo. At its core you create an `XamarinAndroidProject` instance. This can be `XamarinAndroidApplicationProject` or say `XamarinFormsApplicationProject`. @@ -142,8 +160,14 @@ var project = new XamarinAndroidApplicationProject (); You can then Add Items such as source files, images or other files. By default it will create a simple Android App which will include one `MainActivity.cs` and some -standard resources. Properties can be set via the `SetProperty` method. This can be -done globally or for a specific Configuration. +standard resources. If you use one of the variants of the `XamarinAndroidApplicationProject` +like `XamarinFormsApplicationProject` the default project will contain the files +needed for that variant. For example the Xamarin.Forms one will contain `xaml` files +for layout. + +MSBuild Properties can be set via the `SetProperty` method. This can be +done globally or for a specific Configuration. By default the project has a +`DebugConfiguration` and a `ReleaseConfiguration`. ```csharp project.SetProperty ("MyGlobalBoolProperty", "False"); @@ -151,8 +175,9 @@ project.SetProperty (project.DebugConfiguration, "MyDebugBoolProperty", "False") ``` Once you have a project object constructed, you can make use of the `ProjectBuilder` -to build the project. There are two helper methods `CreateApkBuilder` and `CreateDllBuilder` which are available. These will allow you do create a builder to -output an `apk` or in the base of a `Library` project a `dll`. +to build the project. There are two helper methods `CreateApkBuilder` and `CreateDllBuilder` +which are available in the `BaseTest` class. These will allow you do create a builder +to output an `apk` or in the base of a `Library` project a `dll`. You call `CreateApkBuilder` to create the builder then pass the project to the `Build` method. This will build the project. There are other methods such as `Save` and `Install` which can be used to run the various underlying MSBuild targets. NOTE: You should wrap your instances of a `ProjectBuilder` inside a `using` to make sure that the files are cleaned up correctly after the test has run. Tests which fail @@ -169,13 +194,36 @@ projects and will generate .NET based projects. When running under `msbuild` it generate the old style projects. This allows you do write the same test for both types of SDK. +### Device Tests +Device based tests are located in `tests/MSBuildDeviceIntegration/MSBuildDeviceIntegration.csproj`. +These work in a similar fashion to the other MSBuild related tests. The only requirement is +that they need a Device Attached. + +The `DeviceTest` base class provides helper methods which will allow you to run your test application on +the device. It also contains methods for capturing the `logcat` output, the UI and changing users. -### Device Tests -Device based tests are located in `tests/MSBuildDeviceIntegration/MSBuildDeviceIntegration.csproj`. These work in a similar fashion to the other MSBuild -related tests. The only requirement is that they need a Device Attached. +```csharp +[Test] +public void MyAppShouldRun ([Values (true, false)] bool isRelease) +{ + var proj = new XamarinAndroidApplicationProject () { + IsRelease = isRelease, + }; + proj.SetDefaultTargetDevice (); + using (var b = CreateApkBuilder (Path.Combine ("temp", TestName))) { + // Build and Install the app + Assert.True (b.Install (proj), "Project should have installed."); + // Run it + RunProjectAndAssert (proj, b); + // Wait for the app to start with a 30 second timeout + Assert.True (WaitForActivityToStart (proj.PackageName, "MainActivity", + Path.Combine (Root, b.ProjectDirectory, "logcat.log"), 30), "Activity should have started."); + } +} +``` ### On Device Unit Tests From 3a2b58df9ac94543e0ebd3c4b50887101764b4f3 Mon Sep 17 00:00:00 2001 From: Dean Ellis Date: Mon, 13 Mar 2023 12:53:01 +0000 Subject: [PATCH 03/12] Update docs --- Documentation/workflow/UnitTests.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/Documentation/workflow/UnitTests.md b/Documentation/workflow/UnitTests.md index 75a6b697c05..6584fe5d4e8 100644 --- a/Documentation/workflow/UnitTests.md +++ b/Documentation/workflow/UnitTests.md @@ -202,8 +202,17 @@ that they need a Device Attached. The `DeviceTest` base class provides helper methods which will allow you to run your test application on the device. It also contains methods for capturing the `logcat` output, the UI and changing users. +You still use the various `Save` and `Build` methods on the `ProjectBuilder` class to build the app, but +you can also use the `Install` method to install the app on the device or emulator. +The `SetDefaultTargetDevice` method on the `XamarinAndroidApplicationProject` will set the MSBuild `AdbTarget` +property from the `ADB_TARGET` environment variable. This will ensure that the test will use the same device +that the environment wants it to use. The `ADB_TARGET` environment variable can be useful if you are running +on a system which has multiple devices attached. +In the example below the `RunProjectAndAssert` method will call the underlying `Run` target in MSBuild and make +sure it runs successfully. The `WaitForActivityToStart` method is the one which monitors the `adb logcat` output +to detect when the app starts. ```csharp [Test] @@ -213,7 +222,7 @@ public void MyAppShouldRun ([Values (true, false)] bool isRelease) IsRelease = isRelease, }; proj.SetDefaultTargetDevice (); - using (var b = CreateApkBuilder (Path.Combine ("temp", TestName))) { + using (var b = CreateApkBuilder ()) { // Build and Install the app Assert.True (b.Install (proj), "Project should have installed."); // Run it @@ -225,6 +234,11 @@ public void MyAppShouldRun ([Values (true, false)] bool isRelease) } ``` +If you want to check if a ui element was shown you can make use of the `GetUI` method. This returns an +`xml` representation of what is one the screen of the device at the time of the call. You can also call +`ClickButton` to click a specific part of the screen. While the method is called `ClickButton` it actually +sends a `tap` to the screen at a specific point. + ### On Device Unit Tests There are a category of tests which run on the device itself, these tests the runtime From f468c21e9a3fca623931ba399a940cef4eda4645 Mon Sep 17 00:00:00 2001 From: Dean Ellis Date: Mon, 13 Mar 2023 13:41:13 +0000 Subject: [PATCH 04/12] update docs --- Documentation/workflow/UnitTests.md | 25 +++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/Documentation/workflow/UnitTests.md b/Documentation/workflow/UnitTests.md index 6584fe5d4e8..0e853fe9800 100644 --- a/Documentation/workflow/UnitTests.md +++ b/Documentation/workflow/UnitTests.md @@ -239,8 +239,29 @@ If you want to check if a ui element was shown you can make use of the `GetUI` m `ClickButton` to click a specific part of the screen. While the method is called `ClickButton` it actually sends a `tap` to the screen at a specific point. +```csharp +[Test] +public void MyAppShouldRunAndRespondToClick () +{ + var proj = new XamarinAndroidApplicationProject (); + proj.SetDefaultTargetDevice (); + using (var b = CreateApkBuilder ()) { + // Build and Install the app + Assert.True (b.Install (proj), "Project should have installed."); + // Run it + RunProjectAndAssert (proj, b); + // Wait for the app to start with a 30 second timeout + Assert.True (WaitForActivityToStart (proj.PackageName, "MainActivity", + Path.Combine (Root, b.ProjectDirectory, "logcat.log"), 30), "Activity should have started."); + Assert.True (ClickButton ("", "android:id/myButton", "Hello World, Click Me!"), "Button should have been clicked."); + } +} +``` + ### On Device Unit Tests There are a category of tests which run on the device itself, these tests the runtime -behaviour. These run `NUnit` and `XUnit` tests directly on the device. Some of these are located in the runtime itself. We build them within the repo then run the tests on -the device. +behaviour. These run `NUnit` tests directly on the device. Some of these are located in the runtime itself. We build them within the repo then run the tests on the device. They use a custom mobile version of +`NUnit` called `NUnitLite`. For the most part they are the same. + +These tests are generally found in the `tests/Mono.Android-Tests` directory. From 51f011ea6509bb7e51a83a7bd34a395ecbe95194 Mon Sep 17 00:00:00 2001 From: Dean Ellis Date: Thu, 16 Mar 2023 10:29:30 +0000 Subject: [PATCH 05/12] Update the docs --- Documentation/workflow/UnitTests.md | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/Documentation/workflow/UnitTests.md b/Documentation/workflow/UnitTests.md index 0e853fe9800..dac5fb4050a 100644 --- a/Documentation/workflow/UnitTests.md +++ b/Documentation/workflow/UnitTests.md @@ -10,11 +10,11 @@ 1. Build Tests - There are a number of `Build` related tests which test various aspects of the `.NET Android` build system. This includes not only MSBuild Task tests but build integration tests. Since the SDK is consumed from MSBuild, almost all of the unit tests are MSBuild related. + There are a number of `Build` related tests which test various aspects of the .NET Android build system. This includes not only MSBuild Task tests but build integration tests. Since the SDK is consumed from MSBuild, almost all of the unit tests are MSBuild related. 2. Device Tests - Because `.NET Android` has to work on devices, we have a number of + Because .NET Android has to work on devices, we have a number of "Integration" tests which check to make sure a final app will function as expected on an Emulator or Device. Our CI system will test against an Emulator. However the system will pick up the first attached device, so developers can test one physical hardware if needed. @@ -48,15 +48,30 @@ To run ALL the supported Device Integraton tests runs. NOTE: Not all tests work under .NET Android yet. So we need to filter them on the `DotNetIgnore` category. +To run a specific test you can use the `Name` argument for the `--filter`, + +`dotnet-local.sh test bin/TestDebug/net7.0/Xamarin.Android.Build.Tests.dll --filter=Name=BuildBasicApplication` + +To list all the available tests use the `-lt` argument + +`dotnet-local.sh test bin/TestDebug/net7.0/Xamarin.Android.Build.Tests.dll -lt` ### Windows On Windows we can use the same `dotnet-local` script to run the tests just like we do on other platforms. -`dotnet-local.cmd test bin/TestDebug/net7.0/Xamarin.Android.Build.Tests.dll --filter=Category!=DotNetIgnore` +`dotnet-local.cmd test bin\TestDebug\net7.0\Xamarin.Android.Build.Tests.dll --filter=Category!=DotNetIgnore` + +`dotnet-local.cmd test bin\TestDebug\MSBuildDeviceIntegration\net7.0\MSBuildDeviceIntegration.dll --filter=Category!=DotNetIgnore` + +To run a specific test you can use the `Name` argument for the `--filter`, + +`dotnet-local.cmd test bin\TestDebug\net7.0\Xamarin.Android.Build.Tests.dll --filter=Name=BuildBasicApplication` + +To list all the available tests use the `-lt` argument -`dotnet-local.cmd test bin/TestDebug/MSBuildDeviceIntegration/net7.0/MSBuildDeviceIntegration.dll --filter=Category!=DotNetIgnore` +`dotnet-local.cmd test bin\TestDebug\net7.0\Xamarin.Android.Build.Tests.dll -lt` ## Writing Tests From b1c53581504aa9f42caf87d1329b0a208b065d69 Mon Sep 17 00:00:00 2001 From: Dean Ellis Date: Wed, 22 Mar 2023 11:04:51 +0000 Subject: [PATCH 06/12] Add more docs --- Documentation/workflow/UnitTests.md | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/Documentation/workflow/UnitTests.md b/Documentation/workflow/UnitTests.md index dac5fb4050a..2201f0f13b0 100644 --- a/Documentation/workflow/UnitTests.md +++ b/Documentation/workflow/UnitTests.md @@ -279,4 +279,17 @@ There are a category of tests which run on the device itself, these tests the ru behaviour. These run `NUnit` tests directly on the device. Some of these are located in the runtime itself. We build them within the repo then run the tests on the device. They use a custom mobile version of `NUnit` called `NUnitLite`. For the most part they are the same. -These tests are generally found in the `tests/Mono.Android-Tests` directory. +These tests are generally found in the `tests/Mono.Android-Tests` directory and are used to test the +functions of the bound C# API on device. The following is an example. + +```csharp +[Test] +public void ApplicationContextIsApp () +{ + Assert.IsTrue (Application.Context is App); + Assert.IsTrue (App.Created); +} +``` + +Tests in this area are usually located in a directory representing the namespace of the API being tested. +For example the above test exists in the `Android.App` folder, since it is testing the `Android.App.Application` class. From 196dcf956c61a2f763d854983ee8e73d72b8ddc7 Mon Sep 17 00:00:00 2001 From: Dean Ellis Date: Wed, 22 Mar 2023 11:59:57 +0000 Subject: [PATCH 07/12] Add test Count --- Documentation/workflow/UnitTests.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/Documentation/workflow/UnitTests.md b/Documentation/workflow/UnitTests.md index 2201f0f13b0..f77bdcfd163 100644 --- a/Documentation/workflow/UnitTests.md +++ b/Documentation/workflow/UnitTests.md @@ -22,9 +22,10 @@ ## Test Count | Test | Count | -| :---------: | ----: | -| Test | 0 | -| Test | 100 | +| :------------------: | -----: | +| MSBuild Tests | 436 | +| MSBuild Device Tests | 105 | +| On Device Tests | 177 | ## Running Tests From 24bbf8663980863af46e296c32cc52c4d970c892 Mon Sep 17 00:00:00 2001 From: Jonathan Pryor Date: Thu, 13 Apr 2023 15:32:02 -0400 Subject: [PATCH 08/12] Update UnitTests.md --- Documentation/workflow/UnitTests.md | 120 ++++++++++++++++------------ 1 file changed, 68 insertions(+), 52 deletions(-) diff --git a/Documentation/workflow/UnitTests.md b/Documentation/workflow/UnitTests.md index f77bdcfd163..d1b5462eaaf 100644 --- a/Documentation/workflow/UnitTests.md +++ b/Documentation/workflow/UnitTests.md @@ -16,7 +16,7 @@ Because .NET Android has to work on devices, we have a number of "Integration" tests which check to make sure a final app will function as expected on an Emulator or Device. Our CI system will - test against an Emulator. However the system will pick up the first attached device, so developers can test one physical hardware + test against an Emulator. However the system will pick up the first attached device, so developers can test on physical hardware if needed. ## Test Count @@ -30,64 +30,80 @@ ## Running Tests -Running tests in the IDE is currently not supported. +Running tests in an IDE is not currently supported. -After building the repo you can make use of the `dotnet-local` script -to run unit tests against the locally build SDK. The `dotnet-local` script is a wrapper around the custom `dotnet` installation which the -build downloads and installs in `bin/Debug/dotnet` or `bin/Release/dotnet` (depending on your configuration). +After building the repo you can make use of the `dotnet-local.sh` or `dotnet-local.cmd` scripts in the top checkout directory +to run unit tests against the locally build SDK. The `dotnet-local*` scripts are wrappers around the custom `dotnet` installation which the +build downloads and installs into `bin/Debug/dotnet` or `bin/Release/dotnet` (depending on your configuration). ### MacOS/Linux To run ALL the build tests run. -`dotnet-local.sh test bin/TestDebug/net7.0/Xamarin.Android.Build.Tests.dll --filter=Category!=DotNetIgnore` +```sh +./dotnet-local.sh test bin/TestDebug/net7.0/Xamarin.Android.Build.Tests.dll --filter=Category!=DotNetIgnore +``` To run ALL the supported Device Integraton tests runs. -`dotnet-local.sh test bin/TestDebug/MSBuildDeviceIntegration/net7.0/MSBuildDeviceIntegration.dll --filter=Category!=DotNetIgnore` +```sh +./dotnet-local.sh test bin/TestDebug/MSBuildDeviceIntegration/net7.0/MSBuildDeviceIntegration.dll --filter=Category!=DotNetIgnore +``` NOTE: Not all tests work under .NET Android yet. So we need to filter them on the `DotNetIgnore` category. To run a specific test you can use the `Name` argument for the `--filter`, -`dotnet-local.sh test bin/TestDebug/net7.0/Xamarin.Android.Build.Tests.dll --filter=Name=BuildBasicApplication` +```sh +./dotnet-local.sh test bin/TestDebug/net7.0/Xamarin.Android.Build.Tests.dll --filter=Name=BuildBasicApplication +``` To list all the available tests use the `-lt` argument -`dotnet-local.sh test bin/TestDebug/net7.0/Xamarin.Android.Build.Tests.dll -lt` +```sh +./dotnet-local.sh test bin/TestDebug/net7.0/Xamarin.Android.Build.Tests.dll -lt +``` ### Windows -On Windows we can use the same `dotnet-local` script to run the tests +On Windows we can use the `dotnet-local.cmd` script to run the tests just like we do on other platforms. -`dotnet-local.cmd test bin\TestDebug\net7.0\Xamarin.Android.Build.Tests.dll --filter=Category!=DotNetIgnore` +```cmd +dotnet-local.cmd test bin\TestDebug\net7.0\Xamarin.Android.Build.Tests.dll --filter=Category!=DotNetIgnore +``` -`dotnet-local.cmd test bin\TestDebug\MSBuildDeviceIntegration\net7.0\MSBuildDeviceIntegration.dll --filter=Category!=DotNetIgnore` +```cmd +dotnet-local.cmd test bin\TestDebug\MSBuildDeviceIntegration\net7.0\MSBuildDeviceIntegration.dll --filter=Category!=DotNetIgnore +``` To run a specific test you can use the `Name` argument for the `--filter`, -`dotnet-local.cmd test bin\TestDebug\net7.0\Xamarin.Android.Build.Tests.dll --filter=Name=BuildBasicApplication` +```cmd +dotnet-local.cmd test bin\TestDebug\net7.0\Xamarin.Android.Build.Tests.dll --filter=Name=BuildBasicApplication +``` To list all the available tests use the `-lt` argument -`dotnet-local.cmd test bin\TestDebug\net7.0\Xamarin.Android.Build.Tests.dll -lt` +```cmd +dotnet-local.cmd test bin\TestDebug\net7.0\Xamarin.Android.Build.Tests.dll -lt +``` ## Writing Tests This section outlines how to write the unit tests for the various parts of the SDK. -Any new test `class` should derive from `BaseTest` or in the case of Device based tests, `DeviceTest`. These base classes provide additional helper methods to +Any new test `class` should derive from [`BaseTest`](../../src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Utilities/BaseTest.cs) or in the case of Device based tests, [`DeviceTest`](../../src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Utilities/DeviceTest.cs). These base classes provide additional helper methods to create and run the unit tests. They also contain methods to run things like `adb` commands and to auto cleanup the unit tests. They will also capture additional things like screenshots if a test fails. ### Task Tests -Tests which run on MSBuild Tasks are located in the `src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Xamarin.Android.Build.Tests.csproj` project. They should be placed in the `src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Tasks/` folder along with the other tests. +Tests which run on MSBuild Tasks are located in the [`Xamarin.Android.Build.Tests.csproj`](../../src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Xamarin.Android.Build.Tests.csproj) project. They should be placed in the [`Tests/Xamarin.Android.Build.Tests/Tasks`](../../src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Tasks/) folder along with the other tests. -There is an implementation of the `IBuildEngine*` interfaces in the `MockBuildEngine` class. You can use this to mock the MSBuild -runtime and test tasks directly. You might need to create an instance of the `MockBuildEngine` per test since it captures warnings and errors to specific collections provided to the constructor. So if you are testing if a `Task` produces a specific error it will need its own `MockBuildEngine`. Just in case the test is run in parallel. +There is an implementation of the [`IBuildEngine`](https://learn.microsoft.com/en-us/dotnet/api/microsoft.build.framework.ibuildengine?view=msbuild-17-netcore) and related interfaces in the [`MockBuildEngine`](../../src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Utilities/MockBuildEngine.cs) class. You can use this to mock the MSBuild +runtime and test tasks directly. You might need to create an instance of the `MockBuildEngine` per test since it captures warnings and errors to specific collections provided to the constructor. If you are testing if a `Task` produces a specific error it will need its own `MockBuildEngine`. Just in case the test is run in parallel. ```csharp var engine = new MockBuildEngine (TestContext.Out); @@ -114,10 +130,10 @@ where a particular test is failing. If you want to capture warnings and errors you need to provide the `MockBuildEngine` with the appropriate arguments. ```csharp -var errors = new List; -var warnings = new List; -var messages = new List; -var engine = new MockBuildEngine (TestContext.Out, errors: errors, warnings: warnings messages:messages); +var errors = new List (); +var warnings = new List (); +var messages = new List (); +var engine = new MockBuildEngine (TestContext.Out, errors: errors, warnings: warnings messages: messages); ``` You can then check these collections for specific output from the `Task`. @@ -128,10 +144,10 @@ Putting it all together [Test] public void MyTaskShouldSucceedWithNoWarnings { - var warnings = new List; - var messages = new List; - var engine = new MockBuildEngine (TestContext.Out, warnings: warnings); - var task = new MyTask () { + var warnings = new List (); + var messages = new List (); + var engine = new MockBuildEngine (TestContext.Out, warnings: warnings); + var task = new MyTask () { BuildEngine = engine, }; Assert.IsTrue (task.Execute (), "Task should succeed."); @@ -139,19 +155,19 @@ public void MyTaskShouldSucceedWithNoWarnings } ``` -Adding `ITaskItem` properties to the Task can be done just like setting normal properties. This way you +Adding [`ITaskItem`](https://learn.microsoft.com/en-us/dotnet/api/microsoft.build.framework.itaskitem?view=msbuild-17-netcore) properties to the Task can be done just like setting normal properties. This way you can test out all sorts of scenarios. ```csharp [Test] public void MyTaskShouldSucceedWithNoWarnings { - var warnings = new List; - var messages = new List; - var engine = new MockBuildEngine (TestContext.Out, warnings: warnings); - var task = new MyTask () { + var warnings = new List (); + var messages = new List (); + var engine = new MockBuildEngine (TestContext.Out, warnings: warnings); + var task = new MyTask () { BuildEngine = engine, - SomeItem = new TaskItem ("somefile.txt"), + SomeItem = new TaskItem ("somefile.txt"), }; Assert.IsTrue (task.Execute (), "Task should succeed."); Assert.AreEqual (0, warnings.Count, $"Task should not emit any warnings, found {warnings.Count}"); @@ -160,14 +176,14 @@ public void MyTaskShouldSucceedWithNoWarnings ### Build Tests -Tests which need to test the SDK integration are located in the `src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Xamarin.Android.Build.Tests.csproj` project. These types of test do NOT run on a Device. Device tests are slow and expensive to run (time wise). Generally these tests check that apps can build and produce the correct files in the final `apk`. It is also where we add tests for specific user reported issues, for example build errors around non ASCII characters etc. +Tests which need to test the SDK integration are located in the [`Xamarin.Android.Build.Tests.csproj`](../../src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Xamarin.Android.Build.Tests.csproj) project. These types of test do NOT run on a Device. Device tests are slow and expensive to run (time wise). Generally these tests check that apps can build and produce the correct files in the final `apk`. It is also where we add tests for specific user reported issues, for example build errors around non ASCII characters etc. There are other build tests which test other aspects of the SDK. Examples are `tests/CodeBehind/BuildTests/CodeBehindBuildTests.csproj` -Writing a test makes use of the `src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Xamarin.ProjectTools.csproj` API. This API exposes a way to programmatically generating csproj files as well as other application based source code. This saves us from having to have 1000's of csproj files all over the repo. +Writing a test makes use of the [`Xamarin.ProjectTools`](../../src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/) API. This API exposes a way to programmatically generating csproj files as well as other application based source code. This saves us from having to have 1000's of csproj files all over the repo. -At its core you create an `XamarinAndroidProject` instance. This can be +At its core you create an [`XamarinAndroidProject`](../../src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Android/XamarinAndroidProject.cs) instance. This can be `XamarinAndroidApplicationProject` or say `XamarinFormsApplicationProject`. ```csharp @@ -181,7 +197,7 @@ like `XamarinFormsApplicationProject` the default project will contain the files needed for that variant. For example the Xamarin.Forms one will contain `xaml` files for layout. -MSBuild Properties can be set via the `SetProperty` method. This can be +MSBuild Properties can be set via the `SetProperty()` method. This can be done globally or for a specific Configuration. By default the project has a `DebugConfiguration` and a `ReleaseConfiguration`. @@ -190,12 +206,12 @@ project.SetProperty ("MyGlobalBoolProperty", "False"); project.SetProperty (project.DebugConfiguration, "MyDebugBoolProperty", "False"); ``` -Once you have a project object constructed, you can make use of the `ProjectBuilder` -to build the project. There are two helper methods `CreateApkBuilder` and `CreateDllBuilder` +Once you have a project object constructed, you can make use of `ProjectBuilder` +to build the project. There are two helper methods: `CreateApkBuilder()` and `CreateDllBuilder()` which are available in the `BaseTest` class. These will allow you do create a builder to output an `apk` or in the base of a `Library` project a `dll`. -You call `CreateApkBuilder` to create the builder then pass the project to the `Build` -method. This will build the project. There are other methods such as `Save` and `Install` which can be used to run the various underlying MSBuild targets. +You call `CreateApkBuilder()` to create the builder then pass the project to the `Build()` +method. This will build the project. There are other methods such as `Save()` and `Install()` which can be used to run the various underlying MSBuild targets. NOTE: You should wrap your instances of a `ProjectBuilder` inside a `using` to make sure that the files are cleaned up correctly after the test has run. Tests which fail will leave their files on disk to later inspection or archiving. @@ -205,29 +221,29 @@ using (var builder = new CreateApkBuilder ()) { } ``` -When running under .NET the ProjectTools will automatically switch to using SDK style +When running under .NET, ProjectTools will automatically switch to using SDK style projects and will generate .NET based projects. When running under `msbuild` it will generate the old style projects. This allows you do write the same test for both types of SDK. ### Device Tests -Device based tests are located in `tests/MSBuildDeviceIntegration/MSBuildDeviceIntegration.csproj`. +Device based tests are located in [`tests/MSBuildDeviceIntegration`](../../tests/MSBuildDeviceIntegration/MSBuildDeviceIntegration.csproj). These work in a similar fashion to the other MSBuild related tests. The only requirement is that they need a Device Attached. The `DeviceTest` base class provides helper methods which will allow you to run your test application on -the device. It also contains methods for capturing the `logcat` output, the UI and changing users. -You still use the various `Save` and `Build` methods on the `ProjectBuilder` class to build the app, but -you can also use the `Install` method to install the app on the device or emulator. +the device. It also contains methods for capturing the `adb logcat` output, the UI, and changing users. +You still use the various `Save()` and `Build()` methods on the `ProjectBuilder` class to build the app, but +you can also use the `Install()` method to install the app on the device or emulator. -The `SetDefaultTargetDevice` method on the `XamarinAndroidApplicationProject` will set the MSBuild `AdbTarget` +The `SetDefaultTargetDevice()` method on the `XamarinAndroidApplicationProject` will set the MSBuild `AdbTarget` property from the `ADB_TARGET` environment variable. This will ensure that the test will use the same device that the environment wants it to use. The `ADB_TARGET` environment variable can be useful if you are running on a system which has multiple devices attached. -In the example below the `RunProjectAndAssert` method will call the underlying `Run` target in MSBuild and make -sure it runs successfully. The `WaitForActivityToStart` method is the one which monitors the `adb logcat` output +In the example below the `RunProjectAndAssert()` method will call the underlying `Run` target in MSBuild and make +sure it runs successfully. The `WaitForActivityToStart()` method is the one which monitors the `adb logcat` output to detect when the app starts. ```csharp @@ -250,9 +266,9 @@ public void MyAppShouldRun ([Values (true, false)] bool isRelease) } ``` -If you want to check if a ui element was shown you can make use of the `GetUI` method. This returns an -`xml` representation of what is one the screen of the device at the time of the call. You can also call -`ClickButton` to click a specific part of the screen. While the method is called `ClickButton` it actually +If you want to check if a ui element was shown you can make use of the `GetUI()` method. This returns an +XML representation of what is one the screen of the device at the time of the call. You can also call +`ClickButton()` to click a specific part of the screen. While the method is called `ClickButton()` it actually sends a `tap` to the screen at a specific point. ```csharp @@ -280,7 +296,7 @@ There are a category of tests which run on the device itself, these tests the ru behaviour. These run `NUnit` tests directly on the device. Some of these are located in the runtime itself. We build them within the repo then run the tests on the device. They use a custom mobile version of `NUnit` called `NUnitLite`. For the most part they are the same. -These tests are generally found in the `tests/Mono.Android-Tests` directory and are used to test the +These tests are generally found in the [`tests/Mono.Android-Tests`](../../tests/Mono.Android-Tests) directory and are used to test the functions of the bound C# API on device. The following is an example. ```csharp From 59e1ef5706465c457a948f7fad2c84f39c3702cd Mon Sep 17 00:00:00 2001 From: Jonathan Pryor Date: Thu, 13 Apr 2023 16:07:30 -0400 Subject: [PATCH 09/12] Update UnitTests.md --- Documentation/workflow/UnitTests.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Documentation/workflow/UnitTests.md b/Documentation/workflow/UnitTests.md index d1b5462eaaf..7db134ce964 100644 --- a/Documentation/workflow/UnitTests.md +++ b/Documentation/workflow/UnitTests.md @@ -32,7 +32,7 @@ Running tests in an IDE is not currently supported. -After building the repo you can make use of the `dotnet-local.sh` or `dotnet-local.cmd` scripts in the top checkout directory +After building the repo you can make use of the [`dotnet-local.sh`](../../dotnet-local.sh) or [`dotnet-local.cmd`](../../dotnet-local.cmd) scripts in the top checkout directory to run unit tests against the locally build SDK. The `dotnet-local*` scripts are wrappers around the custom `dotnet` installation which the build downloads and installs into `bin/Debug/dotnet` or `bin/Release/dotnet` (depending on your configuration). From de5c87f5de920e50ec3aa1bd78a32653ce6c21c0 Mon Sep 17 00:00:00 2001 From: Jonathan Pryor Date: Fri, 14 Apr 2023 16:09:03 -0400 Subject: [PATCH 10/12] Update UnitTests.md restructure --- Documentation/workflow/UnitTests.md | 361 ++++++++++++++++++---------- 1 file changed, 240 insertions(+), 121 deletions(-) diff --git a/Documentation/workflow/UnitTests.md b/Documentation/workflow/UnitTests.md index 7db134ce964..6161f52b802 100644 --- a/Documentation/workflow/UnitTests.md +++ b/Documentation/workflow/UnitTests.md @@ -1,59 +1,85 @@ # Unit Tests +Unit test infrastructure for .NET Android. -## Tools +# Tools -1. [NUnit](https://github.com/nunit/nunit) -2. [XUnit](https://github.com/xunit/xunit) +Most of our unit test infrastructure uses NUnit. +Many of the BCL tests for Classic Xamarin.Android rely on an xUnit test runner. -## Project Test Types + 1. [NUnit](https://github.com/nunit/nunit) + 2. [xUnit](https://github.com/xunit/xunit) -1. Build Tests +# Project Test Types - There are a number of `Build` related tests which test various aspects of the .NET Android build system. This includes not only MSBuild Task tests but build integration tests. Since the SDK is consumed from MSBuild, almost all of the unit tests are MSBuild related. +There are five types of unit tests within this repo: -2. Device Tests + 1. [MSBuild Integration Tests](#msbuild-integration-tests): integration tests + which exercise an entire MSBuild "pipeline". Most tests are + MSBuild Integration Tests. - Because .NET Android has to work on devices, we have a number of - "Integration" tests which check to make sure a final app will function as expected on an Emulator or Device. Our CI system will - test against an Emulator. However the system will pick up the first attached device, so developers can test on physical hardware - if needed. + 2. [MSBuild Task Unit Tests](#msbuild-task-tests): unit tests which test a + single MSBuild task in isolation, *without* involving the MSBuild engine, + `.targets` files, or anything else. -## Test Count + 3. [Device Integration Tests](#devive-integration-tests): A superset of + MSBuild Integration Tests, these exercise an entire MSBuild "pipeline" + *and also* install and run the resulting app on-device. -| Test | Count | -| :------------------: | -----: | -| MSBuild Tests | 436 | -| MSBuild Device Tests | 105 | -| On Device Tests | 177 | + 4. [On-Device Unit Tests](#device-unit-tests): A set of NUnit unit tests which + run on-device. + 5. [Other Tests](#other-tests): Tests which don't easily fit into the previous + types. -## Running Tests +# Test Count + +| Test | Count | +| :------------------: | ----: | +| MSBuild Tests | 436 | +| MSBuild Device Tests | 105 | +| On Device Tests | 177 | + + +# Running Tests Running tests in an IDE is not currently supported. -After building the repo you can make use of the [`dotnet-local.sh`](../../dotnet-local.sh) or [`dotnet-local.cmd`](../../dotnet-local.cmd) scripts in the top checkout directory -to run unit tests against the locally build SDK. The `dotnet-local*` scripts are wrappers around the custom `dotnet` installation which the -build downloads and installs into `bin/Debug/dotnet` or `bin/Release/dotnet` (depending on your configuration). +After [building the repo](../building), the +[MSBuild Integration Tests](#msbuild-integration-tests), +[MSBuild Task Unit Tests](#msbuild-task-tests), and +[Device Integration Tests](#devive-integration-tests) tests can be run +by using the [`dotnet-local.sh`](../../dotnet-local.sh) and +[`dotnet-local.cmd`](../../dotnet-local.cmd) scripts in the top directory of +the checkout. The `dotnet-local*` scripts are wrappers around a custom +`dotnet` installation which the build downloads and installs into +`bin/Debug/dotnet` or `bin/Release/dotnet` (depending on your configuration). -### MacOS/Linux +## Running on macOS & Linux -To run ALL the build tests run. +On macOS and Windows we can use the `dotnet-local.sh` script to run the tests. + +To run ALL the [MSBuild Integration Tests](#msbuild-integration-tests) *and* +all the [MSBuild Task Unit Tests](#msbuild-task-tests), run: ```sh ./dotnet-local.sh test bin/TestDebug/net7.0/Xamarin.Android.Build.Tests.dll --filter=Category!=DotNetIgnore ``` -To run ALL the supported Device Integraton tests runs. +To run ALL the supported [Device Integration Tests](#devive-integration-tests), run: ```sh ./dotnet-local.sh test bin/TestDebug/MSBuildDeviceIntegration/net7.0/MSBuildDeviceIntegration.dll --filter=Category!=DotNetIgnore ``` +If no Android device is attached, then the emulator will be created. +The `ADB_TARGET` environment variable can be used to explicitly specify which +Android device should be used when running Device Integration Tests. + NOTE: Not all tests work under .NET Android yet. So we need to filter them on the `DotNetIgnore` category. -To run a specific test you can use the `Name` argument for the `--filter`, +To run a specific test you can use the `Name=Value` argument for `--filter`, ```sh ./dotnet-local.sh test bin/TestDebug/net7.0/Xamarin.Android.Build.Tests.dll --filter=Name=BuildBasicApplication @@ -65,19 +91,27 @@ To list all the available tests use the `-lt` argument ./dotnet-local.sh test bin/TestDebug/net7.0/Xamarin.Android.Build.Tests.dll -lt ``` -### Windows +## Running on Windows + +On Windows we can use the `dotnet-local.cmd` script to run the tests. -On Windows we can use the `dotnet-local.cmd` script to run the tests -just like we do on other platforms. +To run ALL the [MSBuild Integration Tests](#msbuild-integration-tests) *and* +all the [MSBuild Task Unit Tests](#msbuild-task-tests), run: ```cmd dotnet-local.cmd test bin\TestDebug\net7.0\Xamarin.Android.Build.Tests.dll --filter=Category!=DotNetIgnore ``` +To run ALL the supported [Device Integration Tests](#devive-integration-tests), runs: + ```cmd dotnet-local.cmd test bin\TestDebug\MSBuildDeviceIntegration\net7.0\MSBuildDeviceIntegration.dll --filter=Category!=DotNetIgnore ``` +If no Android device is attached, then the emulator will be created. +The `ADB_TARGET` environment variable can be used to explicitly specify which +Android device should be used when running Device Integration Tests. + To run a specific test you can use the `Name` argument for the `--filter`, ```cmd @@ -90,20 +124,104 @@ To list all the available tests use the `-lt` argument dotnet-local.cmd test bin\TestDebug\net7.0\Xamarin.Android.Build.Tests.dll -lt ``` -## Writing Tests +# Writing Tests + +How you write tests depends upon the type of test you're writing. + + + +## MSBuild Integration Tests + +MSBuild Integration Tests exercise an entire MSBuild "pipeline", and are +located in +[`src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests`](../../src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests). +These types of test *do not* run on a Device. These tests check that apps can +build and produce the correct files in the final `apk`. It is also where we add +tests for specific user reported issues, for example build errors around non +ASCII characters etc. + +Any new test `class` should derive from +[`BaseTest`](../../src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Utilities/BaseTest.cs). +These base classes provide additional helper methods to create and run the unit +tests. They also contain methods to run things like `adb` commands and to auto +cleanup the unit tests. They will also capture additional things like +screenshots if a test fails. + +Writing a test uses [`Xamarin.ProjectTools`](../../src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/), +which exposes a way to programmatically generating `.csproj` files as well as +other application based source code. This saves us from having to have 1000's +of `csproj` files all over the repo. + +At its core you create an +[`XamarinAndroidProject`](../../src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Android/XamarinAndroidProject.cs) +instance. This can be `XamarinAndroidApplicationProject` or say `XamarinFormsApplicationProject`. + +```csharp +var project = new XamarinAndroidApplicationProject (); +``` + +You can then add Items such as source files, images or other files. By default +it will create a simple Android App which will include one `MainActivity.cs` +and some standard resources. If you use one of the variants of the +`XamarinAndroidApplicationProject` like `XamarinFormsApplicationProject`, the +default project will contain the files needed for that variant. For example the +Xamarin.Forms one will contain XAML files for layout. + +MSBuild Properties can be set via the `SetProperty()` method. This can be +done globally or for a specific Configuration. By default the project has a +`DebugConfiguration` and a `ReleaseConfiguration`. + +```csharp +project.SetProperty ("MyGlobalBoolProperty", "False"); +project.SetProperty (project.DebugConfiguration, "MyDebugBoolProperty", "False"); +``` + +Once you have a project object constructed, you can make use of `ProjectBuilder` +to build the project. There are two helper methods: `CreateApkBuilder()` and +`CreateDllBuilder()` which are available in the `BaseTest` class. +These will allow you do create a builder to output an `apk` or in the base of a +`Library` project a `dll`. +You call `CreateApkBuilder()` to create the builder then pass the project to +the `Build()` method. This will build the project. +There are other methods such as `Save()` and `Install()` which can be used to +run the various underlying MSBuild targets. + +*NOTE*: You should wrap your instances of a `ProjectBuilder` inside a `using` +block to ensure that the files are cleaned up correctly after the test has run. +Tests which fail will leave their files on disk to later inspection or +archiving. + +```csharp +using (var builder = new CreateApkBuilder ()) { + Assert.IsTrue (builder.Build (proj), "Build should have succeeded."); +} +``` + +When running under .NET, `ProjectTools` will automatically switch to using +SDK-style projects and will generate .NET based projects. When running under +`msbuild` it will generate the old style projects. This allows you do write the +same test for both types of SDK. -This section outlines how to write the unit tests for the various parts of the SDK. -Any new test `class` should derive from [`BaseTest`](../../src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Utilities/BaseTest.cs) or in the case of Device based tests, [`DeviceTest`](../../src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Utilities/DeviceTest.cs). These base classes provide additional helper methods to -create and run the unit tests. They also contain methods to run things like -`adb` commands and to auto cleanup the unit tests. They will also capture additional -things like screenshots if a test fails. -### Task Tests + -Tests which run on MSBuild Tasks are located in the [`Xamarin.Android.Build.Tests.csproj`](../../src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Xamarin.Android.Build.Tests.csproj) project. They should be placed in the [`Tests/Xamarin.Android.Build.Tests/Tasks`](../../src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Tasks/) folder along with the other tests. +## MSBuild Task Unit Tests -There is an implementation of the [`IBuildEngine`](https://learn.microsoft.com/en-us/dotnet/api/microsoft.build.framework.ibuildengine?view=msbuild-17-netcore) and related interfaces in the [`MockBuildEngine`](../../src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Utilities/MockBuildEngine.cs) class. You can use this to mock the MSBuild -runtime and test tasks directly. You might need to create an instance of the `MockBuildEngine` per test since it captures warnings and errors to specific collections provided to the constructor. If you are testing if a `Task` produces a specific error it will need its own `MockBuildEngine`. Just in case the test is run in parallel. +MSBuild Task Unit Tests are unit tests which test a single MSBuild task in +isolation, *without* involving the MSBuild engine, `.targets` files, or +anything else. MSBuild Task unit tests are generally in the +[`src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests`](../../src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/) +project. + +There is an implementation of the +[`IBuildEngine`](https://learn.microsoft.com/en-us/dotnet/api/microsoft.build.framework.ibuildengine?view=msbuild-17-netcore) +and related interfaces in the +[`MockBuildEngine`](../../src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Utilities/MockBuildEngine.cs) +class. You can use this to mock the MSBuild runtime and test tasks directly. +You might need to create an instance of the `MockBuildEngine` per test since it +captures warnings and errors to specific collections provided to the constructor. +If you are testing if a `Task` produces a specific error it will need its own +`MockBuildEngine`, just in case the test is run in parallel. ```csharp var engine = new MockBuildEngine (TestContext.Out); @@ -118,16 +236,19 @@ var task = new MyTask () { }; ``` -Then you can `Assert` on the `Execute` method of the task. This will run the task and return a `bool`. +Then you can `Assert` on the `Execute()` method of the task. This will run the +task and return a `bool`. ```csharp Assert.IsTrue (task.Execute (), "Task should succeed."); ``` -NOTE: It is common practice in .NET Android to provide a text description on an `Assert`. This makes it easier to track down -where a particular test is failing. +NOTE: It is common practice in .NET Android to provide a text description on +an `Assert` call. This makes it easier to track down where a particular test +is failing. -If you want to capture warnings and errors you need to provide the `MockBuildEngine` with the appropriate arguments. +If you want to capture warnings and errors you need to provide the +`MockBuildEngine` with the appropriate arguments. ```csharp var errors = new List (); @@ -155,8 +276,9 @@ public void MyTaskShouldSucceedWithNoWarnings } ``` -Adding [`ITaskItem`](https://learn.microsoft.com/en-us/dotnet/api/microsoft.build.framework.itaskitem?view=msbuild-17-netcore) properties to the Task can be done just like setting normal properties. This way you -can test out all sorts of scenarios. +Adding [`ITaskItem`](https://learn.microsoft.com/en-us/dotnet/api/microsoft.build.framework.itaskitem?view=msbuild-17-netcore) +properties to the Task can be done just like setting normal properties. +This way you can test out all sorts of scenarios. ```csharp [Test] @@ -174,77 +296,39 @@ public void MyTaskShouldSucceedWithNoWarnings } ``` -### Build Tests - -Tests which need to test the SDK integration are located in the [`Xamarin.Android.Build.Tests.csproj`](../../src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Xamarin.Android.Build.Tests.csproj) project. These types of test do NOT run on a Device. Device tests are slow and expensive to run (time wise). Generally these tests check that apps can build and produce the correct files in the final `apk`. It is also where we add tests for specific user reported issues, for example build errors around non ASCII characters etc. - -There are other build tests which test other aspects of the SDK. Examples are -`tests/CodeBehind/BuildTests/CodeBehindBuildTests.csproj` + -Writing a test makes use of the [`Xamarin.ProjectTools`](../../src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/) API. This API exposes a way to programmatically generating csproj files as well as other application based source code. This saves us from having to have 1000's of csproj files all over the repo. +## Device Integration Tests -At its core you create an [`XamarinAndroidProject`](../../src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Android/XamarinAndroidProject.cs) instance. This can be -`XamarinAndroidApplicationProject` or say `XamarinFormsApplicationProject`. - -```csharp -var project = new XamarinAndroidApplicationProject (); -``` +Device based tests are located in +[`tests/MSBuildDeviceIntegration`](../../tests/MSBuildDeviceIntegration/MSBuildDeviceIntegration.csproj). +These work in a similar fashion to the other MSBuild related tests. The only +requirement is that they need a Device Attached. -You can then Add Items such as source files, images or other files. By default it -will create a simple Android App which will include one `MainActivity.cs` and some -standard resources. If you use one of the variants of the `XamarinAndroidApplicationProject` -like `XamarinFormsApplicationProject` the default project will contain the files -needed for that variant. For example the Xamarin.Forms one will contain `xaml` files -for layout. +Any new test `class` should derive from +[`DeviceTest`](../../src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Utilities/DeviceTest.cs). +These base classes provide additional helper methods to create and run the unit +tests. They also contain methods to run things like `adb` commands and to auto +cleanup the unit tests. They will also capture additional things like +screenshots if a test fails. -MSBuild Properties can be set via the `SetProperty()` method. This can be -done globally or for a specific Configuration. By default the project has a -`DebugConfiguration` and a `ReleaseConfiguration`. +The `DeviceTest` base class provides helper methods which will allow you to run +your test application on the device. It also contains methods for capturing the +`adb logcat` output, the UI, and changing users. +You still use the various `Save()` and `Build()` methods on the `ProjectBuilder` +class to build the app, but you can also use the `Install()` method to install +the app on the device or emulator. -```csharp -project.SetProperty ("MyGlobalBoolProperty", "False"); -project.SetProperty (project.DebugConfiguration, "MyDebugBoolProperty", "False"); -``` +The `SetDefaultTargetDevice()` method on the `XamarinAndroidApplicationProject` +will set the MSBuild `AdbTarget` property from the `ADB_TARGET` environment +variable. This will ensure that the test will use the same device that the +environment wants it to use. The `ADB_TARGET` environment variable can be +useful if you are running on a system which has multiple devices attached. -Once you have a project object constructed, you can make use of `ProjectBuilder` -to build the project. There are two helper methods: `CreateApkBuilder()` and `CreateDllBuilder()` -which are available in the `BaseTest` class. These will allow you do create a builder -to output an `apk` or in the base of a `Library` project a `dll`. -You call `CreateApkBuilder()` to create the builder then pass the project to the `Build()` -method. This will build the project. There are other methods such as `Save()` and `Install()` which can be used to run the various underlying MSBuild targets. -NOTE: You should wrap your instances of a `ProjectBuilder` inside a `using` to make sure that the files are cleaned up correctly after the test has run. Tests which fail -will leave their files on disk to later inspection or archiving. - -```csharp -using (var builder = new CreateApkBuilder ()) { - Assert.IsTrue (builder.Build (proj), "Build should have succeeded."); -} -``` - -When running under .NET, ProjectTools will automatically switch to using SDK style -projects and will generate .NET based projects. When running under `msbuild` it will -generate the old style projects. This allows you do write the same test for both types -of SDK. - -### Device Tests - -Device based tests are located in [`tests/MSBuildDeviceIntegration`](../../tests/MSBuildDeviceIntegration/MSBuildDeviceIntegration.csproj). -These work in a similar fashion to the other MSBuild related tests. The only requirement is -that they need a Device Attached. - -The `DeviceTest` base class provides helper methods which will allow you to run your test application on -the device. It also contains methods for capturing the `adb logcat` output, the UI, and changing users. -You still use the various `Save()` and `Build()` methods on the `ProjectBuilder` class to build the app, but -you can also use the `Install()` method to install the app on the device or emulator. - -The `SetDefaultTargetDevice()` method on the `XamarinAndroidApplicationProject` will set the MSBuild `AdbTarget` -property from the `ADB_TARGET` environment variable. This will ensure that the test will use the same device -that the environment wants it to use. The `ADB_TARGET` environment variable can be useful if you are running -on a system which has multiple devices attached. - -In the example below the `RunProjectAndAssert()` method will call the underlying `Run` target in MSBuild and make -sure it runs successfully. The `WaitForActivityToStart()` method is the one which monitors the `adb logcat` output -to detect when the app starts. +In the example below the `RunProjectAndAssert()` method will call the +underlying `Run` target in MSBuild and make sure it runs successfully. +The `WaitForActivityToStart()` method is the one which monitors the +`adb logcat` output to detect when the app starts. ```csharp [Test] @@ -266,10 +350,11 @@ public void MyAppShouldRun ([Values (true, false)] bool isRelease) } ``` -If you want to check if a ui element was shown you can make use of the `GetUI()` method. This returns an -XML representation of what is one the screen of the device at the time of the call. You can also call -`ClickButton()` to click a specific part of the screen. While the method is called `ClickButton()` it actually -sends a `tap` to the screen at a specific point. +If you want to check if a UI element was shown you can make use of the +`GetUI()` method. This returns an XML representation of what is one the screen +of the device at the time of the call. You can also call `ClickButton()` to +click a specific part of the screen. While the method is called `ClickButton()` +it actually sends a `tap` to the screen at a specific point. ```csharp [Test] @@ -290,14 +375,37 @@ public void MyAppShouldRunAndRespondToClick () } ``` -### On Device Unit Tests + + +## On-Device Unit Tests + +There are a category of tests which run on the device itself, these test the +runtime behaviour. These run `NUnit` tests directly on the device. Some of +these are located in the runtime itself. We build them within the repo then run +the tests on the device. They use a custom mobile version of `NUnit` called +`NUnitLite`. For the most part they are the same. + +These tests are generally found in: -There are a category of tests which run on the device itself, these tests the runtime -behaviour. These run `NUnit` tests directly on the device. Some of these are located in the runtime itself. We build them within the repo then run the tests on the device. They use a custom mobile version of -`NUnit` called `NUnitLite`. For the most part they are the same. + * [`tests/Mono.Android-Tests`](../../tests/Mono.Android-Tests) + * [`tests/EmbeddedDSOs/EmbeddedDSO`](../../tests/EmbeddedDSOs/EmbeddedDSO) + * [`tests/locales/Xamarin.Android.Locale-Tests`](../../tests/locales/Xamarin.Android.Locale-Tests) -These tests are generally found in the [`tests/Mono.Android-Tests`](../../tests/Mono.Android-Tests) directory and are used to test the -functions of the bound C# API on device. The following is an example. +These tests are run by using the `RunTestApp` target on the appropriate project +file, which includes: + + * `tests/Mono.Android-Tests/Runtime-Microsoft.Android.Sdk/Mono.Android.NET-Tests.csproj` + +For example: + +```zsh +./dotnet-local.sh build -t:RunTestApp tests/Mono.Android-Tests/Runtime-Microsoft.Android.Sdk/Mono.Android.NET-Tests.csproj +``` + +After running the tests, a `TestResult*.xml` file will be created in the +top checkout directory containing the results of the tests. + +The following is an example unit test. ```csharp [Test] @@ -308,5 +416,16 @@ public void ApplicationContextIsApp () } ``` -Tests in this area are usually located in a directory representing the namespace of the API being tested. -For example the above test exists in the `Android.App` folder, since it is testing the `Android.App.Application` class. +Tests in this area are usually located in a directory representing the +namespace of the API being tested. +For example the above test exists in the `Android.App` folder, since it is +testing the `Android.App.Application` class. + + + +## Other Tests + +[`tests/CodeBehind/BuildTests/CodeBehindBuildTests.csproj`](../../tests/CodeBehind/BuildTests/CodeBehindBuildTests.csproj) +is used to test [Layout CodeBehind](../guides/LayoutCodeBehind.md). +If it builds, the test is considered successful, and is built via inclusion +in the [`Xamarin.Android-Tests.sln` project](../../Xamarin.Android-Tests.sln). From db1e5c9a45717223bbcc26b102ce9ced97d863f1 Mon Sep 17 00:00:00 2001 From: Dean Ellis Date: Mon, 17 Apr 2023 15:27:59 +0100 Subject: [PATCH 11/12] Some fixes --- Documentation/workflow/UnitTests.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/Documentation/workflow/UnitTests.md b/Documentation/workflow/UnitTests.md index 6161f52b802..03c53c30383 100644 --- a/Documentation/workflow/UnitTests.md +++ b/Documentation/workflow/UnitTests.md @@ -85,6 +85,13 @@ To run a specific test you can use the `Name=Value` argument for `--filter`, ./dotnet-local.sh test bin/TestDebug/net7.0/Xamarin.Android.Build.Tests.dll --filter=Name=BuildBasicApplication ``` +If the test has arguments then the `=` will not match the name. Instead use the `~`, +this does a "Contains" check for the Name. + +```sh +./dotnet-local.sh test bin/TestDebug/net7.0/Xamarin.Android.Build.Tests.dll --filter=Name~BuildBasicApplication +``` + To list all the available tests use the `-lt` argument ```sh @@ -118,6 +125,13 @@ To run a specific test you can use the `Name` argument for the `--filter`, dotnet-local.cmd test bin\TestDebug\net7.0\Xamarin.Android.Build.Tests.dll --filter=Name=BuildBasicApplication ``` +If the test has arguments then the `=` will not match the name. Instead use the `~`, +this does a "Contains" check for the Name. + +```sh +./dotnet-local.cmd test bin\TestDebug\net7.0\Xamarin.Android.Build.Tests.dll --filter=Name~BuildBasicApplication +``` + To list all the available tests use the `-lt` argument ```cmd @@ -148,7 +162,7 @@ cleanup the unit tests. They will also capture additional things like screenshots if a test fails. Writing a test uses [`Xamarin.ProjectTools`](../../src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/), -which exposes a way to programmatically generating `.csproj` files as well as +which exposes a way to programmatically generate `.csproj` files as well as other application based source code. This saves us from having to have 1000's of `csproj` files all over the repo. From 1a0e71b3fa39b9a337fe3f634aad0193d07194b7 Mon Sep 17 00:00:00 2001 From: Jonathan Pryor Date: Wed, 19 Apr 2023 14:55:44 -0400 Subject: [PATCH 12/12] Update UnitTests.md --- Documentation/workflow/UnitTests.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/Documentation/workflow/UnitTests.md b/Documentation/workflow/UnitTests.md index 03c53c30383..02be11036ca 100644 --- a/Documentation/workflow/UnitTests.md +++ b/Documentation/workflow/UnitTests.md @@ -119,7 +119,10 @@ If no Android device is attached, then the emulator will be created. The `ADB_TARGET` environment variable can be used to explicitly specify which Android device should be used when running Device Integration Tests. -To run a specific test you can use the `Name` argument for the `--filter`, +NOTE: Not all tests work under .NET Android yet. So we need to filter +them on the `DotNetIgnore` category. + +To run a specific test you can use the `Name=Value` argument for the `--filter`, ```cmd dotnet-local.cmd test bin\TestDebug\net7.0\Xamarin.Android.Build.Tests.dll --filter=Name=BuildBasicApplication