Loading ToolBench
A robust FastAPI-based service to detect Tool Poisoning Attacks (SAFE-T1001) and Prompt Injections (SAFE-T1102) with tool authorization and policy management capabilities
This MCP server exhibits significant quality gaps across definition and structure. Of 17 tools examined, descriptions are present but often generic (e.g., 'Get detailed API health status and system metrics' for get_health_status lacks context on WHEN to call it or WHAT downstream decisions it enables). Input schemas are partially documented in the submission but lack formal JSON Schema type declarations visible in the source code. Many parameters lack descriptions entirely (e.g., scan_batch's 'items' array has no item schema defined). Tool naming is verb-initial and generally acceptable, but composition issues are severe: authorization tools (authorize_tool, authorize_tool_with_policy) appear to duplicate functionality with no clear differentiation. Security-sensitive write operations (create_policy, update_policy, delete_policy) lack confirmation/dry-run patterns. Error handling guidance is absent, no recovery hints or actionable error messages documented. Output schemas are not documented anywhere in the provided source.
Add a tool to an existing component policy
Check if a tool is authorized for a component based on policies
Authorize a tool using stored component policy
Create a new access control policy for a component
Delete an access control policy
Get security profile and analysis for a component
Get global security settings and configuration
Output schemas not documented for any tool. LLMs cannot infer what fields they should expect or plan downstream tool calls. This violates the critical requirement that tool output be documented to enable chaining.
Duplicate authorization tools without clear functional differentiation: authorize_tool and authorize_tool_with_policy both authorize tools but accept different parameters. LLMs will conflate them and waste reasoning cycles. Single canonical interface required.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 42 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 30 | - | v1 |
Get detailed API health status and system metrics
Retrieve all access control policies
Get detailed statistics about threat detection and system performance
Get the complete registry of available tools with their metadata
Remove a tool from an existing component policy
Batch scan multiple items for threats
Scan prompt for threats including prompt injection and tool poisoning attacks
Scan MCP tool for poisoning attacks and security vulnerabilities
Update global security settings and configuration
Update an existing access control policy
Destructive operations (delete_policy) lack confirmation/dry-run patterns. Agents can irreversibly delete policies without a chance to review or undo. No error recovery or rollback guidance.
Generic, unhelpful descriptions on read tools (e.g., get_health_status='Get detailed API health status and system metrics', get_policies='Retrieve all access control policies'). Descriptions lack context on WHEN to call them or WHAT decisions they enable. Under 100 chars; most lack the 50-200 char golden zone for LLM comprehension.
scan_batch's 'items' parameter is declared as type 'array' with no item schema. LLMs cannot know what structure each item should have. Is it a string? An object with fields? Undocumented item schema violates constraint declaration.
No error handling guidance documented. Tools offer no recovery hints, actionable error messages, or error classification (retryable vs user-fixable vs fatal). LLMs cannot know whether to retry, ask the user, or give up.
Permission requirements not declared on sensitive tools (create_policy, update_policy, delete_policy, update_global_settings). No scope declarations like 'requires: admin' or 'requires: policy:write'. Cannot configure least-privilege agents.
Many parameters lack descriptions entirely or are too vague. 'items' in scan_batch, 'metadata' in scan_tool, 'context' in multiple tools, no explanation of format, purpose, or expected structure. LLMs cannot infer parameter meaning from names alone.
Tool chaining broken, no evidence that policy creation response returns policy_id or that downstream policy-update tools accept it. Responses likely omit IDs needed for subsequent operations, forcing wasteful lookup calls.
get_tools_registry and get_policies may return unbounded results, risking context window exhaustion. No pagination (limit/offset/next_cursor) or result count cap documented. Large registries will blow token budgets.