Loading ToolBench
AgentNave demonstrates solid tool definition quality with complete schemas, clear descriptions, and proper parameter typing. All four tools have explicit input schemas with type definitions and descriptions. Tool names follow verb_noun convention (start_agent, wait_agent, cancel_agent, describe_provider). Descriptions are substantive (150-400 chars), explaining WHAT the tool does, WHEN to use it, and key constraints. Parameter descriptions are present and specific. However, output schemas lack explicit documentation, responses are TypedDict-based but not formally documented in tool definitions. Error handling is present but could be more prescriptive about recovery paths. Tool annotations are correctly applied to start_agent (destructive_hint=true, read_only_hint=false) but missing from other tools.
Stop a running invocation immediately and return its final result.
Describe a provider's permitted status and supported options.
Start one CLI invocation and return its invocation_id; use wait_agent for the result. AgentNave imposes no runtime deadline; use cancel_agent to stop work explicitly. Provider-native limits still apply. Active invocations do not accept messages. To continue a finished conversation, pass its returned native session_id with a new prompt to a new start_agent call.
Wait for a fixed window of five minutes; completion or an explicit CLI execution blocker returns early. Running responses include the latest public reply tail (at most 1000 characters) and its age. execution_blocked leaves the CLI running: continue waiting or cancel with the same ID. Each blocker category wakes once per invocation. Ordinary tool failures, transient retries, and silence are not proof the task cannot proceed. Finished responses contain the final reply. Wait expiry leaves the invocation running. Call again with the same ID to continue. This is request/response waiting, not a background notification subscription.
Output schemas not formally documented in tool definitions. Response payloads (InvocationPayload, ProviderDescriptionPayload) are TypedDict-based but not exposed in tool registration. LLMs cannot see what fields to expect without reading implementation code.
Tool annotations incomplete. Only start_agent has annotations (destructive_hint, read_only_hint). wait_agent and cancel_agent should declare read_only_hint=true and reversible semantics respectively. describe_provider should declare read_only_hint=true.
Error handling lacks recovery guidance. ToolError exceptions (e.g., 'Unknown invocation_id') do not suggest next steps. Per pattern:recovery-guide, errors should tell LLMs what to do: 'Unknown invocation_id. Verify the ID was returned by start_agent during this session.'
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 64 | 2026-07-28+ | v2 |
describe_provider description is generic (19 chars: 'Describe a provider's permitted status and supported options.'). Should explain WHEN to call it (before start_agent to check availability) and what it reveals (permitted status, supported CLI options).
No batch or multi-invocation tools. If agents frequently manage multiple concurrent subagents, a batch_wait_agents or list_active_invocations tool would reduce round-trips and token waste.