Loading ToolBench
Detect pytest config-leakage that corrupts agent-benchmark grading. Deterministic, no-LLM.
Three tools with clear, domain-specific purposes (pytest config inspection). All have descriptions (100-150 chars, within baseline 34-392 range). Input schemas present with type declarations. However, output schemas are not formally documented in the code, only inferred from return statements. Parameter descriptions are minimal (single short phrases). No error handling guidance for LLMs (e.g., 'path not found' returns a dict but doesn't suggest recovery steps). Tool names are verb-noun style (inspect_workspace, check_addopts, summarize) which is good, but 'summarize' is generic and could be more specific (e.g., 'summarize_workspace_analysis'). No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite all being read-only operations.
Boolean 'is pytest-corrupted?' + reasons for a workspace path.
Check whether running pytest inside `path` will inherit addopts that corrupt agent-benchmark/function grading (rootdir + ini chain + verdict).
One-line actionable summary of an inspect_workspace() result dict.
Output schemas not formally documented. Return types are inferred from code (dicts with 'ok', 'error', 'verdict', etc.) but not declared in tool registration. LLMs cannot reliably extract fields without explicit schema documentation.
Error responses lack recovery guidance. 'path not found' and 'cannot inspect' errors return dicts but do not tell the LLM what to do next (e.g., 'Verify the path exists and is readable'). Agents cannot self-correct.
Parameter descriptions are minimal. 'path' is described only as 'workspace path to inspect', no guidance on format (absolute vs relative), required permissions, or what constitutes a valid workspace.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 60 | 2026-07-28+ | v2 |
Tool 'summarize' is generic and ambiguous. Name does not convey that it specifically summarizes pytest workspace analysis results. Better: 'summarize_workspace_analysis' or 'format_workspace_verdict'.
No tool annotations despite all tools being read-only. Adding readOnlyHint=true would signal to agents that these tools are safe to call without side effects, enabling more aggressive caching and retry strategies.