Loading ToolBench
MCP server for Permit.io authorization and access management
Server provides 7 tools with mixed quality. Tool names follow verb_noun pattern (list_, create_, approve_, deny_, order_). Descriptions are present for all tools but vary in completeness and clarity. Input schemas are visible and properly structured with types and descriptions for most parameters. However, several critical issues limit overall quality: (1) tool #6 and #7 are defined in example files rather than the main server, making their integration unclear; (2) error handling guidance is absent from all tool descriptions; (3) output schemas are not documented; (4) several descriptions contain typos ('existeance' in list_resource_instances, 'Optinoal' in approve_access_request); (5) no per-request logLevel or tool annotations present; (6) descriptions lack guidance on when to use each tool vs. alternatives. Average tool description length is ~100 chars (baseline: 194 chars), well below production standard. Schemas are present but lack output documentation.
Approve an access request.
Create a new access request.
Deny an access request.
List access requests.
Lists the dishes available at a given restaurant along with their prices in dollars. Dishes are only listed when the user has access; otherwise and access request will need to be sent.
Lists resource instances along with their ID and key which can be used as a parameter for tools that required it. It can be used to verify the existeance of a resource instance.
Typos in descriptions undermine clarity: 'existeance' (list_resource_instances), 'Optinoal' (approve_access_request). Typos reduce LLM confidence in parsing tool intent.
No output schemas documented. LLMs cannot infer what fields to expect from responses, forcing them to guess which data to extract for downstream tool calls. This breaks tool composition patterns.
Descriptions lack error handling guidance. When a tool call fails, the LLM cannot determine whether to retry, ask for clarification, or try an alternative tool. No recovery hints provided.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 50 | <=2025-11-25 | v2 |
| 2026-03-09 | D | 51 | - | v1 |
Processes an order for a dish.
Descriptions too short to guide tool selection. Average description length is ~100 chars vs. baseline 194 chars. Minimal context for when to use each tool instead of alternatives.
Tools #6 (list_dishes) and #7 (order_dish) are defined in example/ subdirectory, not in main server.py. Registration mechanism unclear, appears these tools may not be registered with the main MCP server.
Parameter 'resource_instance' accepts both string and integer (union type) but descriptions do not clarify when to use which. LLM guidance missing.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) present. LLMs cannot determine which tools are safe to retry or have side effects without explicit markup.