-
Notifications
You must be signed in to change notification settings - Fork 11.9k
test(@angular/cli): add CI validation for MCP example code #31174
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,75 @@ | ||
# Angular CLI MCP Example Format Specification | ||
|
||
This document specifies the "knowledge packet" format for all markdown examples used by the `find_examples` MCP tool. The goal of this format is to provide a rich, structured, and unambiguous source of information for an AI model, enabling it to understand, apply, and generate code based on modern Angular best practices. | ||
|
||
The format is optimized for both keyword-based full-text search and modern semantic search via a vector database. | ||
|
||
## File Structure | ||
|
||
Each example must be a single `.md` file containing YAML front matter and a structured markdown body. | ||
|
||
### 1. YAML Front Matter | ||
|
||
The front matter provides machine-readable metadata about the example. | ||
|
||
- **`title`** (string, required): A clear, human-readable title for the example (e.g., `Signal Form with Cross-Field Validation`). | ||
- **`summary`** (string, required): A dense, one-sentence, factual description of what the code demonstrates. This field is the primary target for vector database embeddings. | ||
- **`keywords`** (string[], required): A list of keywords and concepts for full-text search. Include APIs, function names, and conceptual terms (e.g., `conditional rendering`). | ||
- **`required_packages`** (string[], optional): A list of NPM packages required for the example to work, excluding `@angular/core`. This is for dependencies that may not be installed (e.g., `@angular/forms`, `@angular/material`). | ||
- **`related_concepts`** (string[], optional): A list of other high-level concepts that are related to this example. | ||
- **`related_tools`** (string[], optional): A list of other MCP tools that are related to this example (e.g., `modernize`). | ||
- **`experimental`** (boolean, optional, default: `false`): When set to `true`, indicates that the example demonstrates an experimental or unstable API that may not be suitable for production use. | ||
|
||
### 2. Markdown Body | ||
|
||
The body provides human-readable and AI-parsable context. It must follow this exact structure and heading order. | ||
|
||
#### `## Purpose` | ||
|
||
- **Content:** A high-level, conceptual explanation of the "why" behind the pattern. It should answer: "What problem does this pattern solve in an application?" | ||
|
||
#### `## When to Use` | ||
|
||
- **Content:** A description of the specific scenarios where this pattern is the preferred best practice. If it supersedes an older pattern, that should be mentioned here (e.g., `@if` vs. `*ngIf`). | ||
|
||
#### `## Key Concepts` | ||
|
||
- **Content:** A bulleted list explaining the main APIs, functions, or decorators demonstrated in the code. Each item should be in the format: `- **`functionName()`:** A brief explanation.` | ||
|
||
#### `## Example Files` | ||
|
||
- **Content:** This section contains the code for the example, structured as a "file-set." | ||
- **Structure:** | ||
- It must start with a brief, one-sentence description of the file-set. | ||
- Each file within the set must be introduced with a `### filename.ext` heading. | ||
- Each file heading must be followed by a one-sentence description of that file's role. | ||
- Each file description must be followed by a fenced code block with the appropriate language identifier (e.g., `typescript`, `html`, `css`). | ||
- **Best Practice:** Examples should be self-contained and runnable, preferably using a `standalone` component. | ||
|
||
#### `## Usage Notes` | ||
|
||
- **Content:** A bulleted list of important details, nuances, or rules about the _internal implementation_ of the code in the `Example Files` section. | ||
|
||
#### `## How to Use This Example` | ||
|
||
- **Content:** A "recipe" that explains how to consume or integrate the example into a larger application. This section is critical for code generation. | ||
- **Structure:** It should contain one or more numbered subheadings (e.g., `### 1. Import the Component`) that provide step-by-step instructions, including code blocks. | ||
- **Flexibility:** The content of this section should be tailored to the type of entity being demonstrated (component, service, pipe, route configuration, etc.). | ||
- **Optional:** This section may not be needed for all examples | ||
|
||
## Search and Embedding Strategy | ||
|
||
This format is designed to be consumed by a sophisticated search system. | ||
|
||
### Keyword Search (FTS) | ||
|
||
The `keywords` field is the primary target for the full-text search engine. The content of the entire document is also indexed. | ||
|
||
### Vector Search (Semantic Search) | ||
|
||
For optimal RAG (Retrieval-Augmented Generation) performance, the document should be chunked before being embedded into a vector database. | ||
|
||
- **Chunking Strategy:** | ||
1. The `summary` field from the front matter should be treated as the primary, document-level chunk and receive its own embedding. | ||
2. Each major markdown section (`## Purpose`, `## When to Use`, `## Key Concepts`, `## Example Files`, `## Usage Notes`, `## How to Use This Example`) should be treated as a separate chunk and receive its own embedding. | ||
- **Metadata:** The full YAML front matter (`title`, `keywords`, `required_packages`, etc.) must be stored as metadata alongside every vector in the database. This allows for powerful post-retrieval filtering and provides the LLM with the full context after a chunk has been identified via semantic search. |
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -0,0 +1,28 @@ | ||||||
{ | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Instead of creating a application and commit it, should we use the current setup we have to generate an application on the fly? This is similar to other validations and it would make it easier to keep the app in code and deps in sync. See: angular-cli/scripts/json-help.mts Lines 34 to 35 in a195db3
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thinking more about this, instead of added another CI step, why not simply add a bazel test? diff --git a/packages/angular/cli/lib/examples/BUILD.bazel b/packages/angular/cli/lib/examples/BUILD.bazel
new file mode 100644
index 000000000..b1f9df0c5
--- /dev/null
+++ b/packages/angular/cli/lib/examples/BUILD.bazel
@@ -0,0 +1,16 @@
+
+filegroup(
+ name = "examples",
+ srcs = glob( ["**/*.md"]),
+)
+
+sh_test(
+ name = "test",
+ srcs = ["//tools:ng_example_db"],
+ data = [":examples", "//tools:ng_example_db"],
+ args = [
+ "$(locations :examples)",
+ "--mode validate-structure",
+ ],
+)
diff --git a/tools/process_examples.mjs b/tools/process_examples.mjs
index 143f8d95a..2939bb9ea 100755
--- a/tools/process_examples.mjs
+++ b/tools/process_examples.mjs
@@ -162,14 +162,12 @@ function parseExampleFile(filePath) {
}
// --- Mode Implementations ---
-
function runValidateStructure(examplesPath) {
console.log('Validating markdown structure of all example files...');
- const exampleFiles = globSync('**/*.md', { cwd: examplesPath });
- for (const file of exampleFiles) {
- parseExampleFile(join(examplesPath, file));
+ for (const file of examplesPath) {
+ parseExampleFile(file);
}
- console.log(`Successfully validated structure of ${exampleFiles.length} files.`);
+ console.log(`Successfully validated structure of ${examplesPath.length} files.`);
}
function runGenerateDb(examplesPath, outputPath) {
@@ -357,7 +355,8 @@ function main() {
},
},
});
- const examplesPath = positionals[0] ?? '.';
+
+ const examplesPath = positionals ?? '.';
const mode = values.mode;
console.log(`Running example processor in '${mode}' mode.`);
@@ -371,7 +370,7 @@ function main() {
runValidateCode(examplesPath);
break;
case 'generate-db':
- runGenerateDb(examplesPath, values.output);
+ runGenerateDb(examplesPath[0], values.output);
break;
default:
throw new Error(
In a follow up we could also update IMO: I think it easier in terms of maintainability and future proofing to have all tests run with a single command and under bazel, instead of having multiple commands |
||||||
"$schema": "./node_modules/@angular/cli/lib/config/schema.json", | ||||||
"version": 1, | ||||||
"newProjectRoot": "projects", | ||||||
"projects": { | ||||||
"validation-harness": { | ||||||
"projectType": "application", | ||||||
"root": "", | ||||||
"sourceRoot": "src", | ||||||
"prefix": "app", | ||||||
"architect": { | ||||||
"build": { | ||||||
"builder": "@angular/build:application", | ||||||
"options": { | ||||||
"outputPath": "dist/validation-harness", | ||||||
"index": "src/index.html", | ||||||
"browser": "src/main.ts", | ||||||
"polyfills": ["zone.js"], | ||||||
"tsConfig": "tsconfig.app.json", | ||||||
"assets": [], | ||||||
"styles": [], | ||||||
"scripts": [] | ||||||
} | ||||||
} | ||||||
} | ||||||
} | ||||||
} | ||||||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
{ | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. NIT: should this has it's own pnpm lock and workspace file, seeing that there are a seperate |
||
"name": "example-validation-harness", | ||
"version": "0.0.0", | ||
"private": true, | ||
"dependencies": { | ||
"@angular/common": "^21.0.0-next", | ||
"@angular/compiler": "^21.0.0-next", | ||
"@angular/core": "^21.0.0-next", | ||
"@angular/forms": "^21.0.0-next", | ||
"@angular/platform-browser": "^21.0.0-next", | ||
"rxjs": "~7.8.0", | ||
"tslib": "^2.3.0", | ||
"zone.js": "~0.15.0" | ||
}, | ||
"devDependencies": { | ||
"@angular/build": "^21.0.0-next", | ||
"@angular/cli": "^21.0.0-next", | ||
"@angular/compiler-cli": "^21.0.0-next", | ||
"typescript": "~5.9.2" | ||
} | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
/** | ||
* @license | ||
* Copyright Google LLC All Rights Reserved. | ||
* | ||
* Use of this source code is governed by an MIT-style license that can be | ||
* found in the LICENSE file at https://angular.dev/license | ||
*/ | ||
|
||
import { Component } from '@angular/core'; | ||
|
||
@Component({ | ||
selector: 'app-root', | ||
imports: [], | ||
template: '', | ||
}) | ||
export class App {} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
<!doctype html> | ||
<html lang="en"> | ||
<head> | ||
<meta charset="utf-8"> | ||
<title>ValidationHarness</title> | ||
<base href="/"> | ||
<meta name="viewport" content="width=device-width, initial-scale=1"> | ||
<link rel="icon" type="image/x-icon" href="favicon.ico"> | ||
</head> | ||
<body> | ||
<app-root></app-root> | ||
</body> | ||
</html> |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
/** | ||
* @license | ||
* Copyright Google LLC All Rights Reserved. | ||
* | ||
* Use of this source code is governed by an MIT-style license that can be | ||
* found in the LICENSE file at https://angular.dev/license | ||
*/ | ||
|
||
import { bootstrapApplication } from '@angular/platform-browser'; | ||
import { App } from './app/app'; | ||
|
||
bootstrapApplication(App).catch((err) => console.error(err)); |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
{ | ||
"extends": "./tsconfig.json", | ||
"compilerOptions": { | ||
"outDir": "./out-tsc/app", | ||
"types": [] | ||
}, | ||
"include": ["src/**/*.ts"], | ||
"exclude": ["src/**/*.spec.ts"] | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
{ | ||
"compileOnSave": false, | ||
"compilerOptions": { | ||
"strict": true, | ||
"noImplicitOverride": true, | ||
"noPropertyAccessFromIndexSignature": true, | ||
"noImplicitReturns": true, | ||
"noFallthroughCasesInSwitch": true, | ||
"skipLibCheck": true, | ||
"isolatedModules": true, | ||
"experimentalDecorators": true, | ||
"importHelpers": true, | ||
"target": "ES2022", | ||
"module": "preserve" | ||
}, | ||
"angularCompilerOptions": { | ||
"enableI18nLegacyMessageIdFormat": false, | ||
"strictInjectionParameters": true, | ||
"strictInputAccessModifiers": true, | ||
"typeCheckHostBindings": true, | ||
"strictTemplates": true | ||
}, | ||
"files": [], | ||
"references": [ | ||
{ | ||
"path": "./tsconfig.app.json" | ||
} | ||
] | ||
} |
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: consider moving this to before the license checks so that it's always executed even when there are license failures, this is useful for automated dependency bumps. Potentially move
Check Package Licenses
as the last step