[feat] enable dart_mcp_server to drive UI and test screenshots

[csells]

to build a UI interactively from a design, an AI agent needs an evaluation function. in this case, the standard MCP server for driving UIs on the web won't work, since Puppeteer requires the HTML DOM, which Flutter Web doesn't provider. we really need the dart_mcp_server to allow the AI agent to drive a Flutter UI such that it can press buttons, enter text, scroll, etc. With that plus screenshots, the AI agent has everything it would need to allow Flutter devs to build Flutter UIs like the AI agents build and iterate on web-based UIs today.

[jakemac53]

Note that the flutter driver tool does now exist (only on the dart dev channel and flutter main channels) - however we ended up disabling screenshots due to the support only existing on the skia renderer which means the majority of platforms it doesn't work by default.

We also likely need to iterate on the flutter driver support based on user feedback.

I am going to leave this issue open as the canonical place to discuss the current support and any improvements that should be made to it.

[csells]

I think the dart mcp server needs a few things to make it a viable option for using an AI agent to iterate on a Flutter app:

• the ability to start and manage the state of a running app, e.g. launch, hot reload, etc.
• the ability to drive the UI via button presses, form filling, etc. so the AI can see the UI in various states and navigate through the app
• the ability to take screenshots so the AI can see what the UI looks like currently and fix UX design deviations or runtime errors
• the ability to see debug output to find and fix runtime errors

It's these capabilities that allow the agent to drive and test and fix a Flutter app rapidly w/o human intervention.

[jakemac53]

Yes those are all likely useful things - the only one we don't have right now is the screenshot support.

The flutter driver stuff (which allows it to drive the UI) also likely needs polish, but I am not 100% clear on the workflows people like to use for this. If you can come up with some clear user journeys for us to test that would be very helpful.

[jakemac53]

See flutter/flutter#170357 for the issue that would unblock screenshots.

[csells]

@jakemac53 the flutter driver stuff is new and super exciting! I'll experiment with what's there. Also excited for screenshots when you can get them.

[jakemac53]

For flutter driver the biggest issue I was running into is getting it to reliably figure out how to select the right thing on the screen. Likely we need some prompting work here, but I don't know enough about flutter driver itself to know the best course of action to suggest in the prompt.

Usually, I find they like to guess at tooltip or text content, and then eventually end up getting the widget tree when that fails. Even after getting the widget tree they sometimes have trouble clicking specific buttons in menu bars.

The other issue I was having is that Gemini 2.5 pro is just too slow based on the number of function calls required to do things, I have a lot better experience with faster but less intelligent models. They might fumble around a bit more but ultimately complete the task much more quickly.

[csells]

For flutter driver the biggest issue I was running into is getting it to reliably figure out how to select the right thing on the screen. Likely we need some prompting work here, but I don't know enough about flutter driver itself to know the best course of action to suggest in the prompt.

Usually, I find they like to guess at tooltip or text content, and then eventually end up getting the widget tree when that fails. Even after getting the widget tree they sometimes have trouble clicking specific buttons in menu bars.

The description for the flutter_driver tool doesn't contain the details necessary for the LLM to know what it can do or how it can do it. I recommend a much more detailed description in the flutter_driver tool. Also, I don't know how the system prompt is created, but I recommend duplicating all of the tool schema information in there, too. I've had very good luck with these two techniques.

[jakemac53]

Yes I agree figuring out the right prompting (via descriptions) is going to improve this a lot. It's figuring out the right prompting that is hard, when I have never used flutter driver myself or have any context into the best way to use it 🤣.

I will try and find somebody with some domain knowledge though to figure that out.

[jakemac53]

Fwiw, I played with this a whole bunch last week, trimmed down the set of available tools some and provided some more instruction in the tool itself, and in general things are working quite reliably. I even managed to live demo it without issues.

I have added a prompt to the MCP server as well, which is what I demo'd. You give it a user journey, and it will try to accomplish that user journey using flutter_driver. Once it has done so, it will write a flutter driver test for you based on the necessary actions it performed, and then also run that test to make sure it works. It works like a charm on simple-ish demos and I am excited to get some real world feedback on it once it hits beta!

I am also going to open up discussions on this repo (well, somebody with access is going to), and then I will convert this issue to a discussion.