Loading ToolBench
Runtime authorization for AI agents — Python SDK. A client library for the Veto authorization API that enables runtime permission checking for AI agent tool calls.
The server defines 11 tools with complete input schemas and descriptions. Naming follows verb_noun convention (authorize, create_agent, list_agents, etc.). Descriptions are present but brief (avg ~60 chars), lacking context on WHEN to use each tool or dependencies between them. Parameter descriptions are minimal, most are 1-2 sentences. Output schemas are not documented in the source; LLMs cannot infer what fields to expect from responses. Error handling is absent from tool definitions. No tool annotations (readOnlyHint, destructiveHint) despite clear risk levels (READ_ONLY, WRITE, DESTRUCTIVE). The authorize tool is well-named and central to the domain, but supporting tools (create_policy, update_policy) lack guidance on rule structure and policy composition.
Check if an agent is authorized to call a tool. This is the primary method — the whole reason Veto exists.
Create a new agent in the Veto system.
Create a new authorization policy for an agent.
Delete an agent by ID.
Delete a policy by ID.
Get a specific agent by ID.
Get a specific policy by ID.
Output schemas not documented. LLMs cannot infer response structure (fields, types, nested objects). Forces agents to guess what data is available after each call.
Tool descriptions are too brief (avg 60 chars, baseline 194 chars). Missing WHEN to use each tool, dependencies, and prerequisites. E.g., create_policy lacks guidance on rule structure and priority semantics.
No tool annotations despite explicit risk levels. Tools marked DESTRUCTIVE (delete_agent, delete_policy) lack destructiveHint. READ_ONLY tools lack readOnlyHint. Agents cannot infer safety implications.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 56 | 2026-07-28+ | v2 |
List all agents in the workspace.
List authorization policies, optionally filtered by agent ID.
Query the audit log with optional filters for agent ID, action, tool name, result, and time range.
Update an existing authorization policy.
Parameter descriptions are minimal (1-2 sentences). 'rules' in create_policy and update_policy lacks format guidance. 'parameters' in authorize lacks structure. LLMs cannot construct valid payloads.
No error handling guidance. Tools lack recovery hints (e.g., 'If agent not found, call list_agents() first'). No categorization of errors as retryable vs. fatal.