Loading ToolBench
A Rust-based authorization and policy enforcement system with benchmarking, CLI tools, and server components for managing tool access control and decision-making
Chaperone is a benchmarking harness for evaluating agent safety, not a production MCP server. The 15 tools in crates/chaperone-bench/src/tools.rs are mock implementations for testing attack scenarios. Tool definitions lack critical production-grade attributes: descriptions are minimal (10-30 chars, well below the 50-200 char LLM-optimized baseline), input schemas are skeletal with no type constraints or enums, no output schemas are documented, and error handling is absent. Parameters lack descriptions entirely (e.g., stripe.refunds.create has 'amount' and 'customer_id' with only type and description field, but no guidance on format, range, or validation). No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite high-risk operations (shell.exec, db.query, stripe.payouts.create). This is research-grade code, not a tool for production agents.
Execute a database query
Send an email
Delete a file from the filesystem
Read a file from the filesystem
Write to a file on the filesystem
Push to a git repository
List HR employees
Descriptions critically short (10-30 chars). Rubric baseline is 50-200 chars for LLM-optimized tools. Examples: 'List Stripe customers' (21 chars), 'Update a Salesforce opportunity' (30 chars). LLMs cannot determine when/why to select tools from such sparse text.
Input schemas incomplete or missing. salesforce.opportunity.update and salesforce.contact.list have empty input objects ({}). No parameter descriptions, no type constraints, no enums. Rubric requires every parameter to have a description explaining what it controls.
No output schemas documented. Rubric baseline: 100% of A+ tools have documented return types. Agents cannot plan downstream calls or extract required fields without knowing what each tool returns.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 38 | 2026-07-28+ | v2 |
List Salesforce contacts
Update a Salesforce opportunity
Execute a shell command
List Stripe customers
Create a payout in Stripe
Create a refund for a Stripe customer
Fetch content from a web URL
Search the web
No tool annotations despite high-risk operations. shell.exec, db.query, stripe.payouts.create, fs.delete are destructive but lack destructiveHint annotation. Agents cannot reason about safety without explicit markers.
No error handling guidance. Rubric requires error responses to tell LLMs what to do next (retryable, user-fixable, fatal). No recovery guides, no categorization, no actionable error messages defined.