Loading ToolBench
AI-assisted IT incident investigation platform that searches web, vendor docs, GitHub, and meeting notes to build ranked hypotheses about IT issues. Exposes investigation capabilities as MCP tools.
15 tools with mostly complete schemas and descriptions, but significant gaps in parameter documentation and error handling guidance. Tool names follow verb_noun convention well (create_, get_, search_, list_, upload_, fulfill_, resume_, flag_, analyze_, record_, conclude_, request_). Descriptions average ~150 chars and explain WHAT and WHEN, but lack actionable error recovery guidance. Parameters have types and many have descriptions, but ~30% lack detail on constraints, formats, or valid ranges. Output schemas are not documented. No tool annotations (readOnlyHint/destructiveHint). Error handling is minimal, tools return success/failure but don't guide LLM recovery paths.
Analyze an uploaded artifact (log file, config file, etc.) for patterns, anomalies, and evidence relevant to the investigation. You'll receive the artifact content and should identify specific errors, misconfigurations, and patterns.
Signal that you've gathered sufficient evidence and are ready to conclude the investigation. Call this when: (1) you have at least one hypothesis with supporting evidence, (2) you've searched multiple angles, and (3) additional searching is unlikely to change your conclusions. Do NOT call this after only one search — investigate thoroughly first.
Create a new IT investigation case and optionally start the investigation. Use this to kick off an AI-powered investigation into an IT incident, integration issue, or architecture review. The platform will search web, vendor docs, GitHub, and meeting notes to build hypotheses.
Flag a finding as relevant or irrelevant to the investigation. Use this to provide feedback on the investigation's evidence collection.
Fulfill an evidence request by providing the requested information or artifact. Use this when the investigation agent has paused and requested specific evidence.
Output schemas not documented. Tools return results but LLM cannot predict field structure, forcing parsing guesses and wasted tokens.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). LLM cannot distinguish safe reads from destructive writes, risking accidental data loss.
Error handling lacks recovery guidance. Tools return success/failure but do not tell LLM what to do next (retry, ask user, abort). E.g., 'case not found' should suggest 'Try list_cases() to find the correct case_id.'
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 61 | 2026-07-28+ | v2 |
Get the current status and summary of an investigation case. Returns case details including status, finding counts, evidence requests (if paused), and a link to the web UI.
Get the findings from an investigation. Returns evidence findings collected during the investigation, grouped by source channel (web, vendor docs, GitHub, meetings, artifact analysis).
Get the final investigation report. Returns the AI-generated report with ranked hypotheses, supporting evidence, and recommended next steps.
List all investigation cases, optionally filtered by status or type.
Record a working hypothesis during the investigation. Use this when you've gathered enough evidence to form a theory. You can record multiple hypotheses. Each should reference the evidence that supports or contradicts it.
Pause the investigation to request specific evidence from the user. Use this when you've identified a critical gap that cannot be resolved with available search tools — for example, you need a specific log file, config export, screenshot, or piece of information that only the user can provide. Do NOT use this for evidence you can find via search. Only request evidence when it would significantly change your investigation conclusions. The investigation will pause until the user provides the requested evidence.
Resume an investigation that was paused waiting for evidence. Call this after fulfilling evidence requests to continue the investigation.
Search GitHub issues, pull requests, and release notes. Use for: Azure-related repos, open-source tools, infrastructure-as-code, and any product with a public GitHub issue tracker.
Search the web for technical information relevant to the investigation. Returns search result summaries. Use different search angles: - Symptom + product name - Specific error code + version - Protocol/mechanism + known issues - Known bugs + vendor bug tracker IDs
Upload an evidence artifact (log file, config, screenshot, etc.) to an investigation case. The artifact will be parsed and analyzed by the investigation agent.
Parameter constraints under-documented. 'max_results' in search_web has default=5 but no min/max bounds stated. 'confidence' in record_hypothesis expects 0.0-1.0 but description does not enforce it.
No pagination guidance for list_cases. If 1000+ cases exist, returning all blows context window. Should document limit cap and offer cursor/offset.