Loading ToolBench
Augur pipeline control — tools for pipeline lifecycle, event injection, state inspection, and control. All Redis I/O is routed through PersistenceManager via context managers that close sockets on exit.
Three tools with basic descriptions but significant gaps in schema completeness and parameter documentation. start_pipeline and stop_pipeline accept array inputs but lack enum constraints or validation guidance. pipeline_status has no input parameters documented. No output schemas are visible in the code. Error handling uses a generic _tool_safe wrapper that returns {"error": str(exc)}, which is better than crashing but lacks actionable recovery guidance. Tool names follow verb_noun convention (start_, stop_, pipeline_status) which is good, but descriptions are terse (under 100 chars) and lack context on when to use each tool or what happens on failure. No tool annotations (readOnlyHint, destructiveHint) despite clear risk levels (WRITE vs READ_ONLY). Security validation exists (_SAFE_LABEL_RE) but is not documented in parameter descriptions, forcing LLMs to guess at constraints.
Return running/exited status, PIDs, and uptime for all tracked processes.
Launch Augur components as subprocesses.
Stop running Augur components.
No output schemas documented. Tools return dicts but LLMs cannot infer field names, types, or structure. pipeline_status returns {"status", "pid", "uptime"} but this is inferred from code comments, not declared in tool definition.
Parameter 'components' in start_pipeline and stop_pipeline lacks enum constraint or validation description. Code enforces COMPONENT_COMMANDS keys but LLMs see only 'array of strings' with no hint of valid values. Defaults to 'all six components' but description says 'all six' while code shows 10+ components.
Destructive operations (start_pipeline, stop_pipeline) lack tool annotations. No destructiveHint or idempotentHint declared. LLMs cannot distinguish safe reads from state-modifying calls without explicit hints.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 43 | <=2025-11-25 | v2 |
Error messages are generic. _tool_safe wrapper returns {"error": str(exc)} with no recovery guidance. If a component name is invalid, LLM gets 'Unknown component: xyz' but no hint to call pipeline_status first or list valid names.
Security validation (_SAFE_LABEL_RE) is not documented in parameter descriptions. LLMs cannot see the regex constraint and may pass invalid component names. Constraint should be explicit in schema or description.