[Testing] Unify Test App infrastructure + VisualUITestBase

[michael-hawker]

Problem

We want our unit tests to not just be able to test .NET code, but also be able to exercise the UI to the extent possible in the UWP/WinUI test app scenarios.

Existing Solution

In the toolkit, we have a helper class called VisualUITestBase which allows a developer writing a unit test with UITestMethod to set the content of the test app temporarily in order to exercise a component, check its layout, and test other parts which may be required for the control to be visualized.

New Solutions

1. Currently a lot of these tests use XamlReader to load XAML from code-behind to create their scenarios. This is a bit cumbersome, especially without the upcoming raw C# string literals. I'd like to investigate using an attribute style solution like we did for the Toolkit UI tests, so loose XAML files can be used to load test scenarios directly within the application.
2. Once we reach this point, we get pretty close to being able to do more complex UI testing, it'd be great if we could try and bridge the complexity gap of the UI Testing systems that are really hard to work with on automation APIs and connecting test runners and the test hosting apps (and being able to run those in the CI across multiple automation framework technologies)... to instead just build on what we know works easily in the unit test structure that exists with MSTest and see if we can provide some basic UI manipulation helpers in order to test more complex scenarios.

Proposal Breakdown

• Independent: Rename CommunityToolkit.Labs.Shared project to CommunityToolkit.Labs.SampleApp.Shared
• Create a new common\Communitytoolkit.Labs.UnitTests.Shared shared project (to appear under the Shared Solution folder)
  • Add this to the root test heads
  • Remove all App.xaml/UnitTestApp.xaml and MainWindow.xaml files from unit test heads
  • Centralize on CommunityToolkit.Labs.UnitTests namespace
  • Use a common App.xaml file to include WinUI 2 and WinUI 3 for all the test heads (just like the individual platform heads do)
  • Abstract the startup for UWP vs. WinUI 3 (see project heads shared App.xaml.cs file)
  • Make sure WINAPPSDK conditional works the same for unit test projects
• Add VisualUITestBase class from Toolkit version
  • Add the new shared unit test project to the experiment shared projects
  • Add 7.1.2 Toolkit references for UWP/WinUI 3 for Visual Tree helpers and the like
• New VisualUITestBase improvements - handled by API Spec for Reunion Deployment Information API microsoft/WindowsAppSDK#156 now
  • Add ability to lookup attribute in TestInitialize in base class
  • Load from page in shared experiment project
  • Auto Dispatch? (all these UI tests had to dispatch to the UI thread to load, so we can automatically handle that I think?).
• Test works in all environments:
  • UWP/WinUI 3 with all solution
  • UWP/WinUI with experiment solution
  • CI
• Investigate helper methods for more 'ui testing' scenarios
  • Since we own the apps and their deployment can we use more restricted capabilities like the InputInjection rescap? And does that work in WinUI 3 as well?
  • Does this work in the CI?
• Use GridSplitter as a test case and see if we can:
  • [ ]~ Test Localization (will be good test for UWP/WinUI 3 and resource loading)~
  • Test a mouse drag (if we get injector working) replicate the UI Test we had for GridSplitter

[michael-hawker]

Figured out my Newtonsoft.Json issue see comment here in microsoft/WindowsAppSDK#59.

Have the shared infrastructure setup, so going to test with bringing in VisualUITestBase and writing some GridSplitter tests. If that works we can snap microsoft/WindowsAppSDK#126 I think before doing the rest?

[michael-hawker]

Built off of microsoft/WindowsAppSDK#156 and tested some of the Input Injector API stuff, looks really promising. Few issues though.

1. Test failures weren't bubbling up, left comment on the PR.

2. WinAppSDK seems slow to load the test content (especially on 1.0.3, updating to 1.1.1 seemed to help). Not sure if there's an issue with the API the CompositionTargetHelper uses or just things saying they're loaded when they're not rendered yet...
   I do see an InvalidCastException when we call the CompositionTargetHelper in WinUI 3...

   System.InvalidCastException
     HResult=0x80004002
     Message=Specified cast is not valid.
     Source=WinRT.Runtime
     StackTrace:
      at WinRT.ExceptionHelpers.<ThrowExceptionForHR>g__Throw|20_0(Int32 hr)
   

   Filed issue: CompositionTarget.Rendering throws InvalidCastException microsoft/microsoft-ui-xaml#7247

3. The docs don't show how to simulate a mouse move, filed an issue here: Input Injection Sample doesn't show how to move mouse points MicrosoftDocs/windows-dev-docs#3887 - would need to figure out what needs to be done for this, but theoretically possible. May be able to ask @shweaver-MSFT for some ideas here depending on how similar it works compared to something like WinAppDriver.

4. Once I updated to WinAppSDK 1.1.1, I saw the touch point, but it was in the wrong spot. So need to follow-up with the WinUI folks on that, currently using this issue to track: InputInjector mouse click error microsoft/microsoft-ui-xaml#10688

5. Finally, if this does seem all viable (it's so close), then would want to design a more fluent API like the toolkit animations to help build up the chain of execution for easier testing. May ask @Sergio0694 for some thoughts on this later.

Have a branch on top of 156 at llama/input-injection, will rebase once that PR is merged.

[Sergio0694]

Happy to help brainstorm a fluent API surface if we get to that point, of course! Ping me any time 😄

[michael-hawker]

Thanks @Sergio0694, yeah I was roughly thinking something like:

await SimulateInput.StartTouch()
                   .MouseDown(coord)
                   .MouseMove(cx, cy, seconds: 0.5, points: 10)
                   .MouseUp()
                   .Execute();

As we need to preface and end the operation with initializing/uninitializing the touch injector code, as well as generate/use a unique pointer id across the instructions (and if we want to use relative positioning between operations keep track of the initial point).

Anyway, bit more investigation to do, as not sure if this is going to run in the CI yet (though even then may have value).