Loading ToolBench
Project-level MCP server configuration demonstrating three transport types (stdio, HTTP, SSE) with multiple MCP servers: filesystem access, GitHub integration, context knowledge base, internal knowledge base, and two custom demo servers (oreilly-cca-mcp Python FastMCP server and cca-study-mcp TypeScript server).
This server exposes 12 tools across two distinct domains (CCA exam study + Managed Agents API). Tool definitions are present with schemas and descriptions, but quality is inconsistent. CCA tools (score_answer, validate_question, lookup_domain_ref) have clear, domain-specific descriptions (80-150 chars) and well-structured input schemas with enums. Managed Agents tools (list_agents, create_session, send_message, etc.) have generic descriptions (40-90 chars) that lack context about when to use them or what happens on error. Parameter descriptions are sparse across both groups, many params lack guidance on format, constraints, or dependencies. Output schemas are not documented anywhere in the source. Error handling is absent: no recovery guidance, no categorization of retryable vs fatal errors, no actionable error messages. The composition is problematic: send_message + wait_for_idle form a required two-step sequence that should be wrapped into a single tool to match the chat data model. No tool annotations (readOnlyHint, destructiveHint) despite clear read/write distinctions. Overall, definitions are functional but fall short of production quality.
Archive a session when the conversation is finished. Frees resources; the session becomes read-only.
Start a new session with a Managed Agent. Returns a session_id — pass it to send_message / wait_for_idle on every subsequent turn.
Get the full configuration of one Managed Agent.
Get a session's current status, usage, and metadata.
Interrupt a running session (agent stops at the next safe point and goes idle).
List Managed Agents available in this workspace. Use name_contains to filter in busy workspaces.
Output schemas not documented. Tools return data but LLMs cannot plan downstream calls or extract fields without trial-and-error. Affects all 12 tools.
Managed Agents tools lack actionable descriptions. 'Get the full configuration of one Managed Agent' (get_agent) does not explain when to call it vs list_agents, what fields are returned, or how to use the result. Descriptions under 50 chars invite misuse.
send_message + wait_for_idle form a required two-step sequence. Agents must call both in sequence (send_message → wait_for_idle) to get a reply. This violates single-responsibility and forces multi-turn reasoning. Should be wrapped into a single send_and_wait_for_reply tool.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 57 | 2026-07-28+ | v2 |
List a session's event log (messages, tool calls, status changes). Use after_id to fetch only new events.
Return the full reference notes for one CCA-F exam domain (D1-D5). Use this to ground a question or an explanation in the domain's core concepts.
Grade a chosen letter (A-D) against a bank question and return whether it is correct plus per-option rationale. Use this to evaluate a learner's answer.
Send a user message to a running session. Returns immediately; call wait_for_idle to get the reply.
Structurally validate a draft CCA-F question against the format and voice rules. Returns a structured list of concrete issues - an empty list means it passed. Use this after authoring a question to check it before delivery.
Block until the agent finishes the current turn (session goes idle) and return its reply text. Call this immediately after send_message. Typical loop per user turn: send_message → wait_for_idle.
No error handling guidance. Tools have no documented error cases, recovery paths, or categorization (retryable vs fatal). An LLM calling create_session with an invalid agent_id gets no hint on what to do next.
No tool annotations. Tools marked as READ_ONLY or WRITE in source comments, but no readOnlyHint/destructiveHint in schema. LLMs cannot infer safety properties and may retry destructive ops (archive_session, interrupt) inappropriately.