DCR: As a developer I would like to have a test client that would allow me to write unit tests for dialogs in isolation

[gabog]

Issue

TestFlow doesn't provide an easy way of write and debug tests for dialogs. Debugging step by step is cumbersome and the stack traces for test failures are hard to track.

Proposed change

Create a DialogTestClient that would allow developers to write unit tests for dialogs and use standard unit testing tools and frameworks.

The new testing client should be hosted in a separate package "Microsoft.Bot.Builder.Testing" that will host this class and the test framework for bots.

This package will be extended over time to include other testing classes and helpers.

Component Impact

• Create new Microsoft.Bot.Builder.Testing to host the classes
• Create new Microsoft.Bot.Builder.Testing.Tests to unit test the new classes

Implementation notes

• 3rd party frameworks to adopt for each platform are:
  • DotNet: XUnit for unit tests, Moq for mocks, FluentAssertions for assertions (to be confirmed)
  • JS: TBD
  • Python: TBD
• Provide parity for SendActivity operations for test flow
• Use TestAdapter if possible but also provide the ability to pass a custom adapter and a custom callback if needed
• Provide the ability to pass a ChannelId to test different rendering conditions (right now TestAdapter passes ChannelId == "tests" which doesn't really allow the dev to test response rendering for different channels)

Customer Impact

NA: new feature

Tracking Status

Dotnet SDK (Tracked by 2028)

• PR
• Merged

Javascript SDK (Tracked by 969)

• PR
• Merged

Python SDK (Tracked by 198)

• PR
• Merged

[dcr]

[sibbl]

I'm not 100% sure if this is the right spot to ask, but is the DialogTestClient here to stay and how will it improve?

I personally prefer the syntax of TestFlow over the API of DialogTestClient for its compactness and readability. The current samples even show some problems, i.e. that you need complex and moreover untested code in your test package to have a usable syntax (client sample with own data structure vs testflow sample without any custom code).

I especially like the ability to send multiple activities at once (i.e. a parsed transcript file) to TestFlow and let it run them. There's also no need to manually deal with GetNextReply (which I need to do when using the DialogTestClient - whereas TestFlow provides some in-built logic and asserts already).

Is it on the roadmap to make the client more usable for devs? Is feedback welcome here or should other issues be opened?

[gabog]

Hi @sibbl, thank you for your comment, I was on vacation and couldn't respond sooner.

Let me address some of your feedback:

• DialogTestClient is here to stay and we will keep expanding it and adding other capabilities based on customer feedback.
• While TestFlow is easier to read, the chained method approach has several limitations:
  • It is harder to debug step by step (you'll need to add a lamda and a break point if you want to step through your test)
  • It doesn't provide meaningful stack traces if your test fails.
  • It makes writing Data Driven Tests harder. With Data Driven Tests you can write tests where you load the data from code (like the example you linked), a transcript file, or any other source while keeping your test logic unchanged.

With DialogTestClient we are trying to let developers write and create tests that align better with well known testing frameworks like XUnit, MSTest, Mocha, etc.

Have you looked at the documentation for Unit Testing bots? There we go from a simple test (that looks similar to the no custom code example you linked above) to more complex XUnit based Data Driven Tests and tests that use mocks.

This project shows some of the features we are considering for future versions, one of them relates to providing an assertion library to make it easier to assert activities. We are still trying to decide what will be addressed next so feel free to add comments to each one of them to help us prioritize what would be more useful to developers. If you have specific asks, post them as a feature request in this site and we will review with the team.

Finally, this draft document provides some background on how we are approaching the testing framework for bots.

Thanks again for your feedback (keep it coming!) and sorry again for not getting back to you sooner.

Gabo

PS: Just in case, we will not be adding more testing features on the upcoming 4.6 release, we are gathering feedback at this point and we will prioritize the new features based on what developers want to see next.