Loading ToolBench
Domain-agnostic artifact judging + create-verify iteration as Model Context Protocol tools. The 'MCP for MCPs' — score any artifact against any rubric, iterate until quality clears a bar.
judge-mcp demonstrates solid tool design with clear naming (verb_noun pattern), comprehensive descriptions (150-400 chars), and well-structured schemas. All 5 tools have input schemas with typed parameters and descriptions. Error handling includes recovery guidance ('not_found' with rubric_id). Strengths: excellent composition (tools chain naturally), idempotent read operations, and domain-specific output structure. Weaknesses: missing output schema documentation in tool descriptions, no parameter constraints (enums/ranges), no tool annotations (readOnlyHint/destructiveHint), and limited error categorization. iterate_until_threshold lacks dry-run/confirmation despite being destructive (calls LLM repeatedly, incurs cost).
Full rubric definition: name, criteria (id, name, weight, guide), optional anchors (calibration: what a high vs low score looks like) and examples (calibration: artifacts with known scores). Returns the rubric dict, or {"error": "not_found", "rubric_id": <id>} when the id is unknown.
Create-verify loop: generate an artifact for the brief, judge it, feed fixes back, regenerate, repeat until the score clears `threshold`, `max_rounds` is hit, or the loop plateaus (to save cost). This is the agentic quality-control primitive. Composes with any other MCP that can call MCPs: gitlaw-mcp answers a question, iterate scores the answer, loops until it's good. Args: brief: what the artifact should accomplish ("Write a 3-line cold email pitching X to Y mentioning Z…"). rubric_id: the rubric to score against (use list_rubrics() to find). threshold: target overall score 0-10 (default 8). Stops as soon as reached. max_rounds: budget cap on iterations (default 3). checks: optional regex deterministic checks (see judge_artifact). model: override the LLM model (default gpt-4o-mini). Returns: { "brief": <input brief>, "threshold": <float>, "rounds": [{"round","artifact","overall","summary","criteria":[...]}, ...], "best": {"round","artifact","overall","summary"}, ← what you ship "reached_bar": <bool — true if best.overall >= threshold> } Note: this tool spends LLM tokens — 1 generate + 1 judge per round, up to max_rounds. For a 3-round run on gpt-4o-mini, ~$0.005 typically.
Score an artifact against a rubric. The anti-vague-feedback tool. Args: artifact: the text being scored (a code snippet, an email, a clause, a design spec — anything domain-relevant for the rubric). rubric_id: a key from list_rubrics(). E.g. "code-review", "legal-clause", "cold-email", "writing-clarity", "resume-bullet". checks: optional list of {"name": str, "pattern": str, "flags": "i"} regex checks that run alongside the LLM judge. Useful for deterministic guarantees ("does it mention DSGVO?"). model: override the default LLM model (gpt-4o-mini). Set via env var JUDGE_MCP_MODEL for the server default. Returns: { "rubric": "<rubric.name>", "overall": <0.0-10.0, weighted mean of criterion scores>, "criteria": [ {"id", "name", "weight", "score" (1-10), "evidence" (≤22 words), "fix" (≤14 words)} ... ], "checks": [{"name", "pass": bool} ...], "summary": "<≤18-word verdict>" } Or {"error": "not_found", "rubric_id": <id>} when the rubric_id is unknown.
Output schemas not documented in tool descriptions. LLMs cannot plan downstream calls or extract fields without knowing response structure. judge_artifact and iterate_until_threshold return complex nested objects (criteria array, checks array, rounds array) but descriptions do not specify field names or types.
No parameter constraints (enums, ranges, patterns). 'model' parameter accepts any string; 'threshold' accepts any float without 0-10 range validation; 'max_rounds' lacks min/max bounds. LLMs can pass invalid values (threshold=999, max_rounds=-1) that fail at runtime.
Missing tool annotations. iterate_until_threshold is destructive (calls LLM, incurs cost, modifies state via generation loop) but lacks destructiveHint. judge_artifact and get_rubric are read-only but lack readOnlyHint. Annotations guide agent planning and safety.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 62 | <=2025-11-25 | v2 |
Discover available rubrics — built-in (from data/rubrics.json) plus any runtime-registered ones from this server instance. Returns: { "rubrics": [ {"id", "name", "source": "builtin"|"runtime", "criteria_count", "has_anchors", "has_examples"} ... ] } Use list_rubrics() first when you don't know what's available, then get_rubric(id) for the full definition before judging.
Add a rubric at runtime for this server instance. Survives only until the process restarts — for permanent rubrics, PR against data/rubrics.json instead. Spec shape: { "name": str, "criteria": [ {"id": str, "name": str, "weight": int, "guide": str}, ... ], "anchors": {"high": str, "low": str} // optional "examples": [{"score": int, "artifact": str, "note": str}, ...] // optional } Returns: {"registered": True, "rubric_id": <id>, "source": "runtime"} or {"error": ..., "message": ...}
iterate_until_threshold lacks confirmation/dry-run pattern. Tool spends LLM tokens and money (~$0.005 per 3-round run) without agent confirmation. No way to preview cost or abort before committing. Agents can accidentally trigger expensive loops.
Error handling lacks categorization. 'not_found' errors are returned but not classified as retryable vs user-fixable. No guidance on whether to retry, ask user, or escalate. register_rubric error responses are generic ('error', 'message') without actionable recovery steps.