MSTest Source Generator: UITestMethodWithXamlPage

[michael-hawker]

Workaround for microsoft/testfx#740 (comment). Also seems like this method should work based on comment by Morten here (Note his solution is v1 source generator, we should build a v2 one.)

Follow-on from #137

Problem

We have to add an explicit dispatch call to all our UI tests in order to get them to work async and wait for UI to load before testing (don't think we can do that without async). And therefore we hit the issue where we can't use the existing [UITestMethod] attribute which should do that for us.

    [TestMethod]
    [TestPage(typeof(PropertySizerTestInitialBinding))]
    public async Task PropertySizer_TestInitialBinding()
    {
        await App.DispatcherQueue.EnqueueAsync(() => {
            // TestPage shouldn't be null here, but we'll do the safer ?. to be sure.
            var propertySizer = TestPage?.FindDescendant<PropertySizer>();

            Assert.IsNotNull(propertySizer, "Could not find PropertySizer control.");

            // Set in XAML Page LINK: PropertySizerTestInitialBinding.xaml#L14
            Assert.AreEqual(300, propertySizer.Binding, "Property Sizer not at expected initial value.");
        });
    }

What we want

    // We're already dispatched to the UI thread here and can write our test either sync or async
    [UITestMethodWithXamlPage(typeof(PropertySizerTestInitialBinding))] // Rename of TestPage attribute
    public void PropertySizer_TestInitialBinding() // or `async Task` should also be supportable if we need it
    {
        // TestPage shouldn't be null here, but we'll do the safer ?. to be sure.
        var propertySizer = TestPage?.FindDescendant<PropertySizer>();

        Assert.IsNotNull(propertySizer, "Could not find PropertySizer control.");

        // Set in XAML Page LINK: PropertySizerTestInitialBinding.xaml#L14
        Assert.AreEqual(300, propertySizer.Binding, "Property Sizer not at expected initial value.");
    }

Solution

Basically the UITestMethodWithXamlPage attribute would generate a regular TestMethod implementation (similar to the example above) with the async Task and DispatcherQueue code we already know works. However, it would also incorporate the GetPageForTest and TestInitialize code currently from VisualUITestBase (added in #137) to remove reflection and setup the test before calling the attributed method the source generator is writing for (UITextMethodWithXamlPage).

    [TestMethod]
    public async Task PropertySizer_TestInitialBinding_Test()
    {
        await App.DispatcherQueue.EnqueueAsync(async () => {
            // Simplifies our reflection heavy initialization in GetPageForTest!
            TestPage = new PropertySizerTestInitialBinding(); // should have diagnostics if this isn't a FrameworkElement/UserControl/Page thing

            // Set content (currently done in TestInitialize, so that can be removed)
            Task result = SetTestContentAsync(TestPage);

            await result;

            if (!result.IsCompletedSuccessfully)
            {
                throw new Exception($"Failed to load page for {TestContext.TestName} with Exception: {result.Exception?.Message}", result.Exception);
            }

            // Call developer code, Note: would need to generate different and await instead if it was an async method.
            PropertySizer_TestInitialBinding();

            TestPage = null; // can be removed from Cleanup method now
        });
    }

Open Questions

• Do we also want our own fixed UITestMethod which does the same without grabbing a page, but encapsulates the need for dispatching to the queue?

[Arlodotexe]

Do we also want our own fixed UITestMethod which does the same without grabbing a page, but encapsulates the need for dispatching to the queue?

You only dispatch to the UI to interact with the UI, which in Labs means Xaml. Since adding this doesn't solve any immediate problems but greatly increases complexity, I'm going to say no.

[michael-hawker]

@Arlodotexe yeah, I'm leaning towards agreeing. In combination with #144, it should hopefully be pretty easy to add a new page to setup scenarios.

I was thinking more for the simpler cases where you just instantiate a control to check automation peers or those sorts of things. (like this one). But in that case we don't need to dispatch again or set content, so the built-in should still be fine.

I think we'll just need to have some good plethora of sample test case scenarios to template off of and we'll be fine.

[mrlacey]

Open Questions
Do we also want our own fixed UITestMethod which does the same without grabbing a page, but encapsulates the need for dispatching to the queue?

I think this would be really useful and support the vision of Labs making it easier to try things out. (Tests are a great way of demonstrating and trying things out.)
Without doing this, it could be interpreted as implying testing doesn't matter or at least isn't as important.
There's an argument that it's "just one line of code" to add the call to EnqueueAsync but it's two with the statement closing and it adds a level of indentation.
Adding a "working" version of UITestMethod would remove some ceremony and make tests easier to write and read.
Being able to easily create a snippet of XAML from a string and not need to have a separate page (with boilerplate code behind?) will greatly improve testability.
This might not need to be something that's there for the initial launch, but I expect its usefulness to quickly become apparent.