Create GHA to run the claude-code-harness

[tgrunnagle]

Creates a github action to run the claude-code-harness against the mcp-optimizer.

Note: still determining how to run the action before merging.

[claude]

Code Review

Overall: Well-structured GitHub Actions workflow with good security practices (pinned action versions with SHA hashes). A few suggestions for improvement:

Issues & Recommendations

Critical

• Line 54-56: Missing error handling. The test harness run should likely have continue-on-error or explicit failure handling
• Line 49: Uses PAT secret but no documentation on required scopes. Add comment specifying required permissions (e.g., # Requires: repo read access)

Improvements

• Line 28: platforms: linux/amd64 - Consider if multi-platform support needed (arm64)
• Line 31: cache-from: type=gha without cache-to - Add cache-to: type=gha,mode=max for better caching
• Line 63: Path pattern logs/*.jsonl may fail silently if no files exist. Consider adding if-no-files-found: warn or error
• Line 54-56: Hard-coded config paths. Consider making these configurable via workflow inputs for reusability

Minor

• Line 34: Comment says "install uv and thv" but only uv is installed. Update comment to "install uv and ToolHive" or just "install dependencies"
• Line 39: Python 3.13 is specified but project requirements should be verified for compatibility

Security

✅ Good use of SHA-pinned actions
✅ Minimal permissions (contents: read)
⚠️ PAT token stored in secrets (unavoidable but ensure it has minimal scopes)

Suggested additions

Consider adding timeout-minutes to the job to prevent hung workflows

[aponcedeleonch]

would be good to call this workflow from code-checks.yml

[aponcedeleonch]

Would be good to call this as a github action instead of cloning the repo completely. Meaning to do something in this action like:

- name: Run Claude Code test harness
  uses: StacklokLabs/[email protected]
  with:
    mcp_severs: ["time", "fetch"]
    cases:
      query: "In what timezone is Mexico?"
      expected: "The timezone in Mexico is foo"
    optimizer_config:
      toolhive_host: "172.17.0.1"