Loading ToolBench
A runnable demo: a Claude agent that triages incidents, drafts RCAs and suggests runbooks against a mock ServiceNow — over the Model Context Protocol (MCP). The same MCP server plugs into Azure AI Foundry Agent Service.
Six tools with clear verb-noun naming and consistent Zod schema definitions. All tools have descriptions (avg 120 chars) and input parameters are typed. However, output schemas are undocumented, responses are free-text strings without structured field definitions. Error handling is basic (isError flag only, no recovery guidance). Missing parameter descriptions for optional fields and no enum constraints on state values despite a fixed set (1,2,3,6,7,8). Tool composition is sound (single responsibility, chainable IDs), but output richness is limited for agent reasoning.
Append a work note (journal entry) to an incident — e.g. a triage summary, RCA draft or recommended runbook.
Fetch the full record for one incident, including its work-note history.
List incidents in the ITSM queue (number, priority, state, summary).
Free-text search across incident summary, description, category and CI.
Find runbook / knowledge-base entries whose symptoms match the query, ranked by relevance. Pass the incident symptoms or description.
Change an incident's state. Valid: 1 New, 2 In Progress, 3 On Hold, 6 Resolved, 7 Closed, 8 Canceled.
Output schemas undocumented. All tools return free-text strings wrapped in {content:[{type:'text',text:...}]} with no structured field definitions. LLMs cannot parse incident numbers, state values, or runbook IDs from responses without additional reasoning.
update_incident_state accepts numeric state (1,2,3,6,7,8) as free-form number, not enum. Description lists valid values but schema does not constrain them. LLM could pass invalid state=5 and receive error 'Invalid state 5' without self-correction guidance.
Error responses lack recovery guidance. Errors return {content:[...],isError:true} with plain text like 'Incident INC0010005 not found.' No suggestion to search_incidents() or list_incidents() first. Agents cannot self-correct.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 60 | 2026-07-28+ | v2 |
Optional parameter 'author' in add_work_note lacks description detail. Current: 'Who is adding the note (defaults to the agent).' Should specify format (name string, email, or ID) and max length.
No pagination support. list_incidents and search_* tools return all results as concatenated text. Large incident queues will blow context window. Missing limit, offset, and total_count fields.