Loading ToolBench
Single tool with minimal definition quality. Tool name 'soc2box_scan' lacks action verb prefix (should be 'scan_soc2_evidence' or similar). Description is adequate (89 chars) but generic. Input schema present with single string parameter 'target', but lacks format constraints, validation rules, or examples. No output schema documented, LLMs cannot predict response structure. No error handling guidance. Tool is read-only but lacks explicit idempotentHint annotation. Overall definition follows basic MCP structure but misses 54-pattern best practices for LLM-optimized tool design.
SOC 2 evidence collector and control tracker, self-hosted. Returns JSON findings.
Tool name lacks action verb. 'soc2box_scan' should start with 'scan_' or 'get_' to signal intent to LLM. Current name is ambiguous, does it scan, analyze, or report?
No output schema documented. Tool returns JSON findings but LLMs cannot predict structure (fields, types, nesting). Forces LLMs to guess at response shape and plan downstream steps blindly.
Input parameter 'target' lacks format constraints. No description of valid paths (absolute vs relative, file vs directory, symlink handling). LLMs will pass arbitrary strings, risking errors.
No error handling guidance. Tool description does not explain failure modes (path not found, permission denied, timeout) or recovery steps. LLMs cannot self-correct on errors.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 50 | 2026-07-28+ | v2 |
Tool lacks idempotentHint annotation. Read-only tools should declare idempotent=true in schema to signal safe retry behavior to agents.