Loading ToolBench
Audit your repos and cloud providers for CI and infrastructure cost leaks. Ships a CLI and an MCP server for AI coding agents (Claude Code, Codex, any MCP host).
costguard has 7 tools with complete input schemas and descriptions. Naming is clear and verb-driven (audit_*, discover_*, plan_*, apply_*, ingest_*). Descriptions are detailed (100-200 chars) and explain WHAT, WHEN, and safety constraints. However, parameter descriptions lack depth, most are single-phrase summaries without format guidance, constraints, or examples. Output schemas are not documented in the tool definitions. Error handling is implicit (no recovery guidance visible). Risk annotations (READ_ONLY, WRITE) are present but not formalized as tool annotations in the MCP schema. Overall: solid foundation, but parameter and output documentation need strengthening.
Apply autofixes to local CI-workflow files for the three gated rules (ci/no-paths-ignore, ci/no-concurrency, ci/no-timeout). Writes files; idempotent; never pushes git. REQUIRES confirmApply:true.
Run read-only GET-only live-site checks against the given URLs (no browser, no form submit, no credential replay). Returns a Findings envelope.
Audit registered workspaces for cloud-cost waste (CI minutes, cron cadence). Read-only. Returns a Findings envelope with totalMonthlyUsd over cost findings. Set all=true for every workspace, or pass workspaces[]; includeSite adds read-only live-site checks.
Detect which cloud providers a directory uses, from config files, dependency names, and env-var NAMES (never values). Read-only.
Reconcile a billing reading the playwriter MCP server returned into a Finding. Unparseable readings become a kind:diagnostic Finding (excluded from totals); never fabricates a number.
Dry-run the autofixers for the given findings: returns unified diffs only and writes NOTHING. Safe preview before apply_fix.
Parameter descriptions lack format constraints and validation rules. E.g., 'directory' in discover_providers has no guidance on absolute vs relative paths, symlink handling, or max depth. 'urls' in audit_site lacks protocol requirements (http/https only?) or count limits.
Output schemas are not documented in tool definitions. Callers cannot predict the structure of 'Findings envelope' or what fields 'totalMonthlyUsd' contains. This forces LLMs to guess downstream field names and risks broken tool chains.
Risk annotations (READ_ONLY, WRITE) are metadata in the tool registry but not exposed as MCP tool annotations (readOnlyHint, destructiveHint). LLM clients cannot see these hints and may misclassify tool safety.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 58 | 2026-07-28+ | v2 |
Plan a live billing check for a provider. Returns the API-first decision and a consent notice; for a browser-fallback provider it emits a READ-ONLY Playwright snippet (for the playwriter MCP server) ONLY with confirmLive:true. costguard never drives the browser itself.
apply_fix requires confirmApply:true but no error guidance is visible for when confirmApply is false or missing. LLMs need explicit recovery instructions: 'Set confirmApply to true to apply fixes, or call plan_fix first to preview changes.'
plan_live_checks and ingest_live_reading form a multi-step workflow but lack chaining documentation. No guidance on what fields plan_live_checks returns that ingest_live_reading expects, or how to handle Playwright failures.