Loading ToolBench
Policy enforcement and observability platform for agentic AI systems with tool invocation control, approval workflows, and audit logging
OCPA provides 3 tools with basic structure but significant quality gaps. All tools have explicit registration with ToolMetadata in example_app/agent.py, which is good. However, input parameter descriptions are minimal or missing, output schemas are not documented, and descriptions lack depth for LLM decision-making. The server uses a custom registration pattern (register_tool with ToolMetadata) rather than MCP tool schema, tool definitions are inferred from Python function signatures and metadata objects, not from explicit JSON Schema.
Create a support ticket.
Fetch a customer's profile (read-only).
Rotate an API key for a customer.
Input parameter descriptions missing or minimal. 'customer_id' parameter in get_customer_profile and rotate_api_key is described only as 'The customer ID to fetch profile for' and 'The customer ID whose API key to rotate', these are tautological. No guidance on format, constraints, or valid ranges. The 'title' parameter in create_ticket has no description in the schema at all.
Output schemas not documented. Tools return Python dictionaries (get_customer_profile returns {'customer_id', 'name', 'segment', 'flags'}, rotate_api_key returns {'api_key', 'status'}, create_ticket returns {'ticket_id', 'title', 'status'}), but there is no formal output schema definition. LLMs cannot predict return structure or plan chained calls.
Tool definitions are inferred from Python function signatures and ToolMetadata objects, not from explicit MCP tool schemas. This server uses a custom registration pattern incompatible with standard MCP tool definitions.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 52 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 38 | - | v1 |
No error handling guidance. If get_customer_profile receives an invalid customer_id or the backend fails, the tool implementation in tools.py has no error recovery hints. The LLM receives no guidance on whether to retry, ask the user, or treat as fatal.
rotate_api_key is a high-risk tool but lacks dry-run or confirmation flow. The description notes side_effect_level='high' and data_classes=['secrets'], but there is no confirmation_request pattern or preview step before key rotation occurs.
Approval system exists (ocpa/approval.py) but is not integrated into tool invocations. The require_approval and request_approval functions exist but are never called in example_app/agent.py during invoke_tool. Approval gates are unenforced.
Tool descriptions are below baseline length and lack LLM-optimized context. 'Fetch a customer's profile (read-only).' is 47 chars; best practice is 50 - 200 chars with context on when to call vs similar tools. 'Create a support ticket.' is 24 chars, below the 50-char minimum for useful LLM decision-making.
No pagination or result limits documented. If get_customer_profile returns nested objects or lists (e.g., 'flags' array), there is no limit, next_cursor, or total_count. Large result sets would blow context windows.