Loading ToolBench
Agent-run has 11 tools with schemas present but significant quality gaps. Tool naming is inconsistent (start, resume, cancel, steer, answer lack clear verb-noun patterns; list_agents and transcript are better). Parameters lack descriptions in many tools (e.g., capacity_order.action, doc.topic have minimal guidance). Output schemas are undocumented across all tools. Error handling is absent, no recovery guidance, no categorization. Security concerns: start accepts 'orchestrator' and 'account' objects without clear sanitization guidance. Composition is weak: start bundles runtime+model+profile+task+workdir (5 concerns); no clear chaining between tools (e.g., start returns agent_id, but resume/cancel/steer/answer don't document what agent_id format to expect).
Provides an answer to a pending agent question
Cancels an active agent session
Queries or orders capacity for agent execution
Retrieves documentation for a topic
Retrieves execution limits and constraints
Lists agent sessions with optional filtering and pagination
Lists available models across configured runtimes
Resumes an existing agent session by its agent ID
start tool description is 280+ chars of confusing PostToolUse binding language ('Automatic PostToolUse hook binding requires a direct, host-visible mcp__agent_run__start...'), not LLM-optimized. Should be 50-150 chars explaining: starts a new agent session with runtime, model, and task. Binding details belong in implementation docs, not the tool description.
9 of 11 tools have descriptions under 50 characters (resume, cancel, steer, answer, models, limits, doc, capacity_order, list_agents). Baseline is 194 chars avg. These are too brief to guide LLM selection. E.g., 'Resumes an existing agent session by its agent ID' lacks context: when should the LLM call this vs start? What does 'resume' mean operationally?
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields. E.g., start returns agent_id (inferred from resume/cancel/steer/answer params), but no schema declares this. list_agents returns sessions with pagination, but structure is undocumented.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 41 | 2025-06-18+ | v2 |
Starts a new agent session with the specified runtime, model, and profile. Automatic PostToolUse hook binding requires a direct, host-visible mcp__agent_run__start or mcp__agent-run__start call; do not wrap or nest start inside functions.exec, a shell call, another tool, or any other indirect invocation when automatic binding is expected. If a direct call is unavailable, pass the current session identity in orchestrator; otherwise delivery remains bound:false and no completion notice will arrive automatically.
Sends steering input to an active agent session
Retrieves the transcript of an agent session
Parameter descriptions are minimal or missing. E.g., capacity_order.action is 'Capacity action (query or order)', what does 'order' do? What are valid values? doc.topic is 'Documentation topic', what topics exist? list_agents.after_revision is 'Return sessions after this revision', what format is revision? These force LLMs to guess.
No error handling guidance. Tools declare WRITE risk but provide no recovery hints. E.g., if start fails (invalid runtime, model not found, workdir inaccessible), what should the LLM do? Retry? Call models first? Ask the user? No guidance.