Evidence-first game operations incident investigation agent with read-only MCP tools.
Five tools with complete schemas and descriptions, but descriptions are generic and lack LLM-optimized guidance. All tools are read-only with clear naming (verb_noun pattern). Parameters have types and descriptions, but lack constraints (enums, ranges, format specs). Output schemas are not documented. Error handling is minimal, responses use generic {'ok': True/False} pattern without recovery guidance. No tool annotations (readOnlyHint, etc.). Composition is sound: tools chain well (metric_name flows through multiple tools), but descriptions don't explain when to call each tool vs. alternatives.
Compare current and baseline cohorts with deterministic metric code and significance.
Detect statistically supported changes; the LLM is not allowed to invent anomaly scores.
Draft a human-review report and validate that every evidence reference exists.
Read metric formulas, ownership, event sources, and data-quality notes.
Run one validated SELECT/CTE query against allowlisted public analytics tables.
Output schemas not documented. LLMs cannot plan downstream calls or extract fields without knowing response structure. Tools return {'ok': True, ...} but the '...' is undocumented.
Descriptions lack LLM-optimized guidance. Generic docstrings (avg ~60 chars) do not explain WHEN to call each tool vs. alternatives, or what prerequisites exist. E.g., 'compare_cohorts' doesn't say 'Call after get_metric_definition to understand metric structure.'
No parameter constraints (enums, ranges, formats). 'row_limit' accepts 1-500 but description doesn't state bounds. 'z_threshold' is numeric but no range guidance. LLMs may pass invalid values.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 58 | 2026-07-28+ | v2 |
Error handling is minimal. All errors return {'ok': False, 'error': str(exc)} without recovery guidance. LLM cannot determine if error is retryable, user-fixable, or fatal.
No tool annotations. Tools lack readOnlyHint, destructiveHint, or idempotentHint metadata. MCP spec 2026-07-28 supports tool annotations for safety and planning, not used here.