45 specialized judges that evaluate AI-generated code for security, cost, and quality
Single tool 'judges_evaluate' has critical deficiencies. Tool name lacks action verb ('judges_evaluate' is unclear, should be 'evaluate_code' or similar). Input schema is present but sparse: only two parameters with minimal descriptions. No output schema documentation. Description is present (48 chars) but vague and lacks WHEN to use, WHAT it returns, or HOW to interpret results. No error handling guidance. Parameter descriptions are present but lack format constraints, ranges, or valid value enums. The tool appears to invoke 45 judges but returns structure is undocumented, LLM cannot know what to expect from the output, limiting downstream chaining and reasoning. Risk classification as READ_ONLY is noted but tool lacks idempotent/readonly annotations in the schema. Schema is inferred from vscode-extension/package.json rather than explicit MCP server code, reducing confidence in the definition.
Evaluate code for security, cost, and quality issues using 45 judges
Tool name lacks clear action verb. 'judges_evaluate' does not start with a standard verb (get_, create_, update_, search_, evaluate_, analyze_). Should be 'evaluate_code' or 'analyze_code' to match pattern:tool naming conventions.
Tool description is only 48 characters: 'Evaluate code for security, cost, and quality issues using 45 judges'. This is below baseline (p10=34, p90=392 chars; midpoint ~194). Lacks context on WHEN to use, WHAT judges are invoked, WHAT the output structure is, and HOW to interpret results. LLM cannot reason about when to call this vs alternatives or how to chain results.
Output schema is completely undocumented. Tool invokes '45 judges' but no response schema is visible. LLM does not know what fields to expect (e.g., does it return a list of findings, a JSON report, a SARIF object, a summary score?). This breaks pattern:tool and pattern:response-shaper, no way for LLM to extract relevant data or chain to downstream tools.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-21 | F | 36 | 2026-07-28+ | v2 |
| 2026-03-09 | D | 52 | - | v1 |
Parameter 'language' lacks value constraints. Description is only 63 chars and lists examples ('typescript, javascript, python, go, rust, java') as free-form text, not as an enum. Per pattern:constrained-input, enums prevent hallucinated values. LLM may pass invalid language strings like 'ts' or 'py' instead of canonical names.
Parameter 'code' has minimal description (21 chars: 'Source code to evaluate'). Lacks guidance on max length, encoding, or file size limits. LLM may pass arbitrarily large files, causing timeouts or memory exhaustion. Per review:param-validation-rules, should state 'Max length: X chars; supported encodings: UTF-8'.
No error handling guidance documented. If a language is unsupported, code is malformed, or evaluation times out, no recovery path is given. Per pattern:recovery-guide, errors should guide LLM: 'Unsupported language. Try one of: typescript, javascript, python, go, rust, java'.
Tool marked READ_ONLY but no idempotentHint or readOnlyHint annotation in schema. Per current spec (2026-07-28), tool annotations should be present in the schema. This prevents agents from reasoning about retry safety and state modification risk.
Tool definition inferred from vscode-extension/package.json rather than explicit MCP server source. No visible server.ts or index.ts in provided code showing tool registration, schema, or handler. Cannot verify actual input/output structures implemented by the MCP server.