Loading ToolBench
MCP server + execution metrics recorder for agent sessions: real-time cost/token metrics and a full Context Window Explorer breakdown, over a real MCP handshake.
Strong tool definitions with excellent descriptions (avg 250+ chars, well above 194-char baseline) and clear WHEN-to-use guidance. All 8 tools have input schemas with type declarations. However, output schemas are entirely undocumented, LLMs cannot plan downstream calls or extract fields. No tool annotations (readOnlyHint/destructiveHint). Parameter descriptions are thorough but lack explicit constraints (enums, ranges, formats). Error handling descriptions exist but lack recovery guidance (e.g., 'Returns [] for unknown session' tells LLM nothing about what to do next). Tool composition is sound (single responsibility, clear naming), but missing pagination guidance for list-returning tools.
The ordered sequence of tool calls for one session -- each entry {name, args, status}. Use to replay what the agent actually did, in execution order; use get_tool_metrics for aggregate counts. session_id: id from get_recent_sessions / record_session. Returns [] for an unknown or non-owned session.
Ordered, categorized breakdown of everything that entered ONE session's context window. Each block (system prompt, tool specs, injected context, user turns, reasoning, tool calls/results, final answer) is marked user-visible vs invisible overhead, with cumulative character-based token estimates against the model's real window size. Use for context-window composition analysis; use get_session_metrics for exact provider token usage. The estimates here are character-based, not exact Bedrock counts. session_id: id from get_recent_sessions / record_session. Returns [] for an unknown or non-owned session, or one recorded without the optional context_blocks field.
Estimated USD cost as a float. Pass session_id for one session's cost, or period_seconds for the summed cost of your sessions in the last N seconds. Give exactly one; non-owners are scoped to their own sessions. These are estimates from token counts and a static price table, not billed amounts. session_id: optional session id. period_seconds: optional lookback window in seconds (e.g. 86400 for the last day). Returns 0.0 for an unknown or non-owned session_id, and for a period with no matching sessions.
Output schemas completely undocumented. Tools return complex nested structures (e.g., get_session_metrics returns {error, metadata, per_prompt_tokens, latency, cost}) but LLMs have no schema to plan downstream calls or extract fields. This forces agents to guess field names and types.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). record_session is marked WRITE but lacks destructiveHint; get_* tools lack readOnlyHint. LLMs cannot infer retry safety or side-effect scope without explicit annotations.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 66 | <=2025-11-25 | v2 |
List recent sessions, newest first: [{session_id, prompt, model_id, created_at, ...}]. Call this first to discover session_ids for the other get_* tools. Non-owners see only their own sessions; the owner token sees all. limit: max rows to return (default 10), newest first.
Full metrics for ONE recorded session: metadata plus per-prompt tokens, latency, and cost. Use when you have a session_id and need exact provider usage numbers for that session. For the block-by-block token composition of the context window use get_context_timeline instead; for a cost total across many sessions use get_cost_estimate. session_id: the opaque, case-sensitive id returned by record_session or listed by get_recent_sessions. Returns {"error": "session not found"} if the id is unknown or not owned by the caller (the two are deliberately indistinguishable).
Per-turn input/output token and latency breakdown for one session, in turn order. Use to see how token usage grew turn-by-turn within a session; use get_session_metrics for session totals. session_id: id from get_recent_sessions / record_session. Returns [] for an unknown or non-owned session.
Tool-call counts grouped by status (ok / error), for one session or aggregated. Use for a quick success/failure summary; use get_agent_trace for the ordered call list. Non-owners only ever see their own sessions; the owner token sees everyone's. session_id: optional. Omit for the aggregate across all your sessions; pass an id for just that session.
Append ONE agent execution's metrics to this server's store; returns the new session_id. This is how a remote agent gets its own runs into the server, rather than only being able to query what the server owner recorded locally. Attributed to the connected identity: your Google account if you signed in via /auth/login, or owner=None for the owner token. NOT idempotent -- each call mints a new session_id. Never updates or deletes an existing session; the get_* tools read what this writes. prompt: the user prompt that started the run. model_id: the provider model identifier, e.g. "anthropic.claude-3-5-sonnet-20241022-v2:0". loop_result: the run's token / latency / trace payload. Required keys: input_tokens (int), output_tokens (int), total_tokens (int), latency_ms (float), turns (list). Optional: trace (list), context_blocks (list -- omit if you don't have per-block context data; get_context_timeline needs it). - each `turns` item: input_tokens (int), output_tokens (int), latency_ms (float); optional cache_read_input_tokens / cache_write_input_tokens (int, default 0). - each `trace` item: tool (str), args (dict), status (str); optional latency_ms (float, default 0), timestamp (float epoch seconds, default record time). - each `context_blocks` item: category (str), label (str), char_count (int), token_estimate (int); optional turn_n (int or null -- null for a pre-conversation block), status (str), content (str).
Error handling lacks recovery guidance. Descriptions state 'Returns [] for unknown session' or 'Returns 0.0 for unknown session_id' but do not tell LLM what to do next (e.g., 'Try get_recent_sessions() to discover valid session IDs'). Agents cannot self-correct.
Pagination not addressed. get_recent_sessions accepts 'limit' but no offset/cursor. If 100+ sessions exist, LLM cannot iterate through all. Description should state 'Returns newest first; use limit to paginate' and clarify if total_count is returned.
record_session's loop_result parameter is a dict with nested required/optional keys, but no formal schema or enum constraints. LLMs cannot validate structure before calling. Should document required keys (input_tokens, output_tokens, total_tokens, latency_ms, turns) and optional keys (trace, context_blocks) with explicit type constraints.