Loading ToolBench
An MCP server that provides governance, policy evaluation, and audit logging for MCP tool calls. Acts as a gateway to manage agent access to MCP servers with policy-based authorization and comprehensive audit trails.
WhisOS Gateway defines 22 tools with consistent verb-noun naming (call_tool, list_agents, create_policy, etc.). All tools have descriptions and input schemas with type definitions. However, descriptions are often generic and lack LLM-optimized guidance on WHEN to use each tool vs. alternatives. Output schemas are not documented, callers cannot predict response structure. Error handling is minimal; no recovery guidance or actionable error messages. Parameter descriptions exist but lack format constraints, ranges, or dependency documentation. The governance domain is well-structured (agents, servers, policies, rules, audit), but tool composition could be tighter, e.g., call_tool combines policy evaluation, authorization, and audit logging without clear separation of concerns.
Execute a tool on an MCP server with policy evaluation and audit logging. Validates agent authorization, checks policy rules, and logs all tool invocations.
Create a new agent with an auto-generated API key.
Create a new policy for governing tool access.
Create a new policy rule to allow or deny tool access.
Register a new MCP server (stdio or HTTP transport).
Delete an agent and all associated policy rules.
Delete a policy and all associated rules.
Output schemas not documented. Tools return results but callers cannot predict response structure (fields, types, pagination). LLMs cannot plan downstream tool calls or extract required IDs for chaining.
Descriptions lack LLM-optimized guidance. Most descriptions (e.g., 'List all agents in the system') are generic and do not explain WHEN to use this tool vs. alternatives, what prerequisites exist, or what the agent should do with the result.
No error handling guidance. Tools do not document what errors can occur, whether they are retryable, or what the LLM should do next. E.g., 'agent not found' vs. 'policy evaluation failed' vs. 'server unavailable' all return generic errors.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 54 | 2026-07-28+ | v2 |
Delete a policy rule.
Delete an MCP server and all associated tools and policy rules.
Discover all available tools on an MCP server and register them.
Retrieve a specific agent by ID.
Retrieve a specific policy by ID with its rules.
Retrieve a specific policy rule by ID.
Retrieve a specific MCP server by ID.
List all agents in the system with pagination support.
List all policies with pagination support.
List all MCP servers with pagination support.
Query audit events with filtering and pagination.
Update an existing agent's properties.
Update an existing policy's properties.
Update an existing policy rule.
Update an existing MCP server's configuration.
Parameter descriptions lack format constraints and ranges. E.g., 'limit' and 'offset' have no min/max bounds; 'token' has no format hint; 'transport' enum is present but other enums (e.g., 'effect' in create_rule) lack explanation of when to use each value.
call_tool combines multiple concerns (policy evaluation, authorization, audit logging, tool invocation) without clear separation. This violates single-responsibility principle and makes it harder for LLMs to reason about failure modes and recovery.