Loading ToolBench
Run GitHub Copilot CLI behind a local MCP server through ACP.
Three tools with clear names and reasonable descriptions, but schemas lack rigor. ask_copilot has a well-structured input schema with 5 parameters, each typed and described (prompt, context, freshSession, allowTools, model). copilot_session_status and copilot_reset_session have empty input schemas ({}), which is acceptable for read-only/diagnostic tools. However, NO output schemas are documented anywhere in the source, LLMs cannot plan downstream actions without knowing what fields to expect. Descriptions are adequate (50-150 chars) but lack actionable guidance on when to use each tool or how to handle errors. Parameter descriptions are present but generic; e.g., 'freshSession' says 'If true, use a disposable isolated ACP session' but does not explain the tradeoff (isolation vs. context retention) or when an agent should prefer one mode. No error handling guidance, no recovery hints, no examples of expected output structure.
Ask GitHub Copilot a question with optional context, model selection, and session control.
Reset the persistent ACP session, clearing all state and context.
Get diagnostic information about the current persistent ACP session.
No output schemas documented. LLMs cannot infer what fields ask_copilot, copilot_session_status, or copilot_reset_session return, blocking downstream tool composition and forcing agents to guess at response structure.
Descriptions lack actionable guidance. 'Get diagnostic information about the current persistent ACP session' does not explain what fields are returned, when to call it, or how to interpret results. Descriptions should be 50 - 200 chars and answer WHAT, WHEN, and WHAT-TO-DO-WITH-RESULT.
No error handling guidance. If ask_copilot fails due to auth, network, or invalid prompt, the tool returns an error but does not tell the LLM what to do next (retry? ask user? call a different tool?). Errors should include recovery hints.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 50 | <=2025-11-25 | v2 |
Parameter descriptions are generic. 'allowTools' says 'If false, hard block tool/MCP usage; if true (default), allow tools per permissionPolicy' but does not explain the security or performance implications, or when an agent should set it to false.
No idempotency or side-effect documentation. ask_copilot with freshSession=false retains context across calls (stateful), but the description does not warn that repeated calls with the same prompt may produce different results due to accumulated context. Agents need to know which calls are safe to retry.