Loading ToolBench
MCP server providing AI agent observability — tracing, cost tracking, performance monitoring, and audit trails
Server has 6 tools with complete Zod schemas and descriptions. Naming is verb-forward (trace_, track_, log_, get_, detect_) and mostly clear. Descriptions are present (avg ~120 chars) but lack actionable context for LLM selection. Parameters are typed and constrained (enums, min/max), but descriptions are sparse (avg ~50 chars), many lack guidance on format, dependencies, or when to use. Output schemas are not documented; responses are JSON text blobs without field-level documentation. Error handling is minimal, no recovery guidance, no categorization of retryable vs fatal errors. Security: session_id is passed as a parameter (not ideal for sensitive tracing), and no mention of permission gates or audit logging beyond the tool itself.
Flag unusual patterns in agent behavior: cost spikes, high error rates, latency issues, infinite loops, and token explosions.
Get cost breakdown across multiple sessions, grouped by model, provider, tool, or session. Supports time range filtering.
Get a comprehensive cost and performance summary for an agent session, including token usage, tool call stats, and model breakdown.
Log an MCP tool call with performance data including latency, success/failure, and error details.
Log an agent action with metadata for audit trails and debugging. Supports tool calls, LLM requests, decisions, and errors.
Track token usage and cost for an LLM call. Auto-calculates cost from built-in pricing table if per-token costs are not provided.
Output schemas not documented. Tools return JSON text blobs without field-level documentation. LLMs cannot plan downstream calls or extract structured data reliably.
Parameter descriptions are sparse (avg 50 chars). Many lack format guidance, constraints, or dependencies. E.g., 'timestamp' says 'ISO 8601 timestamp (defaults to now)' but doesn't explain what happens if omitted or how it's used in queries.
No error handling guidance. Tools return JSON responses but do not document what errors are possible, whether they are retryable, or what the LLM should do next. E.g., what if session_id doesn't exist?
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 52 | 2026-07-28+ | v2 |
session_id passed as parameter in every tool. For sensitive audit/tracing operations, session context should be injected server-side, not exposed as a parameter that enters logs and prompt history.
get_cost_report and get_session_summary lack pagination. No mention of result limits, cursors, or how to handle large datasets. Returning thousands of cost entries could exhaust context.