From beb06b24e1b1adda2a3fba6b4dd6b6e7b5f53eed Mon Sep 17 00:00:00 2001 From: Karan Gathani Date: Wed, 1 Oct 2025 20:20:57 -0700 Subject: [PATCH 1/5] Add AI-Assisted Testing documentation page Introduces a new documentation page explaining how to use the `shiny add test` command for AI-powered test generation in Shiny for Python. Updates the navigation in _quarto.yml to include the new section and page. --- _quarto.yml | 3 + docs/ai-assisted-testing.qmd | 122 +++++++++++++++++++++++++++++++++++ 2 files changed, 125 insertions(+) create mode 100644 docs/ai-assisted-testing.qmd diff --git a/_quarto.yml b/_quarto.yml index cfb4a24b..9f1f95d2 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -285,6 +285,9 @@ website: contents: - docs/unit-testing.qmd - docs/end-to-end-testing.qmd + - section: "🤖 __AI-Assisted Testing__" + contents: + - docs/ai-assisted-testing.qmd - section: "🏗️ __Extending__" contents: - docs/custom-component-one-off.qmd diff --git a/docs/ai-assisted-testing.qmd b/docs/ai-assisted-testing.qmd new file mode 100644 index 00000000..a4dd6aa3 --- /dev/null +++ b/docs/ai-assisted-testing.qmd @@ -0,0 +1,122 @@ +--- +title: AI-Assisted Testing with Shiny for Python +description: Learn how to leverage AI to automatically generate tests for your Shiny applications using the `shiny add test` command. +editor: + markdown: + wrap: sentence +--- + + +Testing web applications can be time-consuming and complex, but what if AI could help you write tests automatically? Shiny for Python now includes an AI-powered testing feature that analyzes your app and generates comprehensive test files for you! + +## What is `shiny add test`? + +The `shiny add test` command is a built-in CLI tool that uses AI to read your Shiny application code and automatically generate test files. + +## Getting Started + +Before using this feature, you'll need: + +1. **Shiny for Python** installed (with testing dependencies) +2. **An API key** from one of the supported AI providers: + - **Anthropic** (Claude) - Set `ANTHROPIC_API_KEY` environment variable + - **OpenAI** (ChatGPT) - Set `OPENAI_API_KEY` environment variable + +## Basic Usage + +### Example 1: Simple Interactive Mode + +The easiest way to get started is to run the command without any arguments: + +```bash +shiny add test +``` + +This launches an interactive prompt that will: +1. Ask for your app file path (defaults to `app.py`) +2. Suggest a test file name (e.g., `test_app.py`) +3. Generate tests using Anthropic's Claude by default + +### Example 2: Specify Your App File + +If you have a specific app file, provide it directly: + +```bash +shiny add test --app app_dashboard.py +``` + +The command will automatically suggest `test_app_dashboard.py` as the output file. + +### Example 3: Full Control with Custom Paths + +Want to specify both the app file and test file name? + +```bash +shiny add test --app app_dashboard.py --test-file test_app_dashboard.py +```bash +shiny add test --app app.py --test-file test_my_app.py +``` + +**Important:** Test file names must start with `test_` (pytest convention). + +### Example 4: Using Different AI Providers + +Choose your preferred AI provider with the `--provider` flag: + +**Using OpenAI:** +```bash +shiny add test --app app.py --provider openai +``` + +### Example 5: Selecting a Specific Model + +Want to use a particular AI model? Use the `--model` flag: + +**With Anthropic:** +```bash +shiny add test --app app.py --model sonnet +``` + +**With OpenAI:** +```bash +shiny add test --app app.py --provider openai --model gpt-5-mini +``` + +## What Happens Behind the Scenes? + +When you run `shiny add test`, the AI: + +1. **Reads your app code** - Analyzes the structure, UI components, and server logic +2. **Identifies test scenarios** - Determines what needs testing (inputs, outputs, interactions) +3. **Generates test code** - Creates comprehensive test cases with assertions +4. **Writes the test file** - Saves it with proper pytest formatting + +## Running Your Generated Tests + +Once the test file is created, run it with pytest: + +```bash +pytest test_app.py +``` + +For more verbose output: + +```bash +pytest test_app.py -v +``` + + +::: callout-tip +### Tip: Review and Customize +The AI generates solid test foundations, but always review and customize tests to match your specific requirements. +::: + + +## Why Use AI-Assisted Testing? + +- **Saves Time**: Generates tests in seconds instead of hours +- **Comprehensive Coverage**: AI identifies test cases you might miss +- **Best Practices**: Follows pytest conventions and Shiny testing patterns + + +Start testing smarter, not harder! From 739f895bab54b9c54ad829f9ef4d91059862f1c5 Mon Sep 17 00:00:00 2001 From: Karan Gathani Date: Wed, 1 Oct 2025 21:02:22 -0700 Subject: [PATCH 2/5] Remove AI-Assisted Testing section from sidebar The AI-Assisted Testing section header was removed from the website sidebar in _quarto.yml, but its content file remains listed. This streamlines the sidebar organization. --- _quarto.yml | 2 -- 1 file changed, 2 deletions(-) diff --git a/_quarto.yml b/_quarto.yml index 9f1f95d2..88efbcb9 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -285,8 +285,6 @@ website: contents: - docs/unit-testing.qmd - docs/end-to-end-testing.qmd - - section: "🤖 __AI-Assisted Testing__" - contents: - docs/ai-assisted-testing.qmd - section: "🏗️ __Extending__" contents: From 188342cbafa752afeb301081bda25cac71bf68eb Mon Sep 17 00:00:00 2001 From: Karan Gathani Date: Wed, 1 Oct 2025 21:11:14 -0700 Subject: [PATCH 3/5] Add section on test generation output in docs Added a new section describing the output users will see after generating a test, including example output and next steps. This helps clarify the workflow and expectations for users running the test generation command. --- docs/ai-assisted-testing.qmd | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/docs/ai-assisted-testing.qmd b/docs/ai-assisted-testing.qmd index a4dd6aa3..df357e47 100644 --- a/docs/ai-assisted-testing.qmd +++ b/docs/ai-assisted-testing.qmd @@ -82,6 +82,23 @@ shiny add test --app app.py --model sonnet shiny add test --app app.py --provider openai --model gpt-5-mini ``` +## What You'll See After Generation + +When your test is generated, you'll see output similar to this: + +**Example output:** +``` +🤖 Generating test using anthropic provider... +LLM token usage and cost: +Anthropic (claude-sonnet-4-20250514): 12.3k input, 2.1k output | Cost $0.0845 | Time taken: 8.45s + +✅ Test file generated successfully: test_app.py + +Next steps: +- Run pytest test_app.py to run the generated test +- Review and customize the test as needed +``` + ## What Happens Behind the Scenes? When you run `shiny add test`, the AI: From 80c2c3c8fa03f0fa39cb8b45c45e26bff57c1537 Mon Sep 17 00:00:00 2001 From: Karan Gathani Date: Thu, 2 Oct 2025 12:16:59 -0700 Subject: [PATCH 4/5] Update page title in AI-assisted testing docs --- docs/ai-assisted-testing.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ai-assisted-testing.qmd b/docs/ai-assisted-testing.qmd index df357e47..7275350d 100644 --- a/docs/ai-assisted-testing.qmd +++ b/docs/ai-assisted-testing.qmd @@ -1,5 +1,5 @@ --- -title: AI-Assisted Testing with Shiny for Python +title: AI-Assisted Testing description: Learn how to leverage AI to automatically generate tests for your Shiny applications using the `shiny add test` command. editor: markdown: From 3a57d1c4c4002031e01f34b2f5b51fa19f565d46 Mon Sep 17 00:00:00 2001 From: Karan Gathani Date: Thu, 2 Oct 2025 12:20:22 -0700 Subject: [PATCH 5/5] Fix code block formatting in documentation --- docs/ai-assisted-testing.qmd | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/ai-assisted-testing.qmd b/docs/ai-assisted-testing.qmd index 7275350d..de76569a 100644 --- a/docs/ai-assisted-testing.qmd +++ b/docs/ai-assisted-testing.qmd @@ -53,6 +53,8 @@ Want to specify both the app file and test file name? ```bash shiny add test --app app_dashboard.py --test-file test_app_dashboard.py +``` + ```bash shiny add test --app app.py --test-file test_my_app.py ```