Loading ToolBench
MCP server to validate agent health under AOS discipline — catch agents that silently fail in production.
Three tools with clear naming and reasonable descriptions, but parameter documentation is sparse and output schemas are not formally documented. Tool names follow verb_noun convention well (scan, audit, self_test). Descriptions are present but brief (under baseline 194 chars on average). Parameters lack rich type constraints (enums, patterns, min/max bounds). Output is structured JSON but schema not formally declared in code. Error handling is minimal, no recovery guidance or categorization. Overall: decent foundation but needs parameter enrichment and formal output schema documentation.
Audit all agents under target_dir and return aggregate health report.
Scan a single AOS agent by tool_id under target_dir and return health score.
Self-connectivity check — returns sample scan JSON for MCP wiring verification.
Output schemas not formally documented. Tools return structured dicts but no JSON Schema or docstring defines expected fields, types, or required keys for callers.
Parameter descriptions lack actionable constraints. 'target_dir' has no mention of path format, required permissions, or resolution precedence (env var fallback). 'tool_id' says 'Four-digit' but no regex pattern or validation guidance.
No error handling guidance. If tool_id is invalid or target_dir does not exist, no recovery advice is offered. Error messages should tell LLM what to do next (e.g., 'Try aos_agent_health_audit to discover valid tool_ids').
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 58 | 2026-07-28+ | v2 |
Parameter descriptions are generic and too brief. 'target_dir' description (43 chars) is under baseline 72 avg param annotation length. 'mock' descriptions (43-53 chars) lack context on when/why to use mock mode or how it affects results.
Tool descriptions are present but brief. 'aos_agent_health_scan' (118 chars) is below baseline 194. 'aos_agent_health_audit' (106 chars) is very brief. Descriptions should explain WHEN to use each tool (e.g., scan for single agent validation; audit for fleet health).
No enum constraints on 'tool_id'. Description says 'Four-digit agent id (e.g. "1066")' but tool_id is a free-form string. No validation that it is actually 4 digits. Requires LLM to guess format or rely on example.
No pagination or result limiting documented. 'aos_agent_health_audit' returns all agent reports in a list. For large fleets, this could overwhelm context. No limit or cursor parameter offered.
'mock' parameter default behavior is inconsistent. 'aos_agent_health_scan' and 'aos_agent_health_audit' default to False; 'aos_agent_health_self_test' defaults to True. No description explains why or when to use mock=True in production scanning.