Loading ToolBench
Status and observability for AI agents. Tracks agent registration, status updates, errors, tool calls, embeddings, and triggers alerts.
MCPolly defines 14 tools with structured schemas and basic descriptions. Most tools have clear names and documented input parameters using schemars. However, there are significant gaps: (1) Output schemas are not documented anywhere, responses are unstructured JSON strings with no defined structure; (2) Many descriptions are generic and lack actionable context for when/why to call each tool; (3) Parameter descriptions are sparse or missing constraints; (4) No enum constraints for state values despite explicit enums in descriptions (e.g., 'Valid states: starting, running, warning, error, completed, offline, paused, errored' should be enforced); (5) Error handling is minimal, no recovery guidance or classification; (6) No security annotations despite destructive operations (delete_embeddings); (7) Tool composition issues: register_agent and spawn_agent do overlapping work; (8) Missing idempotency guarantees on state mutations. Naming is generally good (verb_noun convention followed), and input schemas are properly defined with schemars, placing this in the C+/B- range. The server is functional but lacks production-grade polish.
Check if a stop request has been issued for an agent
Delete all embeddings for a specific source
Get the recent activity timeline for a specific agent including status updates and errors
Retrieve error logs for a specific agent
Get the current status and details of a specific agent
List all registered agents and their current status
Report an error from an agent. Records the error and triggers configured alerts.
No output schemas documented for any tool. Tools return unstructured JSON strings (e.g., register_agent returns {"id": ..., "name": ..., "created": ...} but this structure is invisible to LLMs). Agents cannot plan downstream calls or extract data reliably.
State/severity enums not enforced in schema. post_status description lists 8 valid states but schemars defines 'state' as a plain string. post_error describes severity as 'error, warning, or critical' but no enum constraint. LLMs will generate invalid values.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 57 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 10 | - | v1 |
Post a status update for an agent. Valid states: starting, running, warning, error, completed, offline, paused, errored
Record a tool invocation by an agent for observability and tracing
Register an AI agent with MCPolly. Returns the agent ID needed for subsequent calls. Idempotent on name.
Search indexed documents using semantic similarity
Request spawning of a new agent task
Request an agent to stop execution
Index and chunk markdown content for semantic search
Destructive tool (delete_embeddings) lacks security annotations and confirmation mechanism. No idempotency guarantee, no audit logging metadata, no dry-run option. Pattern:confirmation-request not implemented.
Weak error handling. Responses are unstructured JSON with 'error' key in some cases (e.g., register_agent returns {"error": "..."} on validation failure), but no error classification (retryable vs fatal), no recovery guidance, no invalid value echoing. LLMs cannot self-correct.
List tool (list_agents) has no pagination parameters (limit, offset, cursor) and no documented result cap. If many agents exist, response could explode context window. Also missing total_count or next_cursor in (undocumented) output.
Tool composition issues: register_agent and spawn_agent both create agents; unclear which to use when. post_status, post_error, and get_agent_activity all deal with agent state but lack clear separation. No guidance in descriptions.
Polling pattern (check_stop_request) is not idiomatic and output schema undefined. No guidance on polling interval, timeout, or response format. Modern MCP favors event-driven or direct return values.
Parameter constraints missing. search_embeddings 'top_k' accepts Option<i64> with no bounds (could be negative, zero, or 1M). source_type filter is freeform string with examples 'prd, design' but no enum. update_embeddings lacks source_type guidance matching search_embeddings filter values.
Descriptions are generic and lack actionable context. Many tools (get_agent_activity, list_agents, spawn_agent) describe WHAT but not WHEN to call or what the result means. No dependency hints (e.g., 'call register_agent first').
No idempotency guarantees documented. Stateful operations (post_status, post_error, post_tool_call, update_embeddings) do not state if they are safe to retry. Agents may avoid retrying failures, reducing reliability.