Agentic workflow replay & audit with OTel GenAI compatibility
The 'scan' tool has a basic schema with two parameters (target, max_tokens) but suffers from critical definition gaps. The tool description is minimal (58 chars) and lacks context on when/why to use it, what it returns, or how errors are handled. Parameter descriptions are present but generic. No output schema is documented. The tool appears to be a wrapper around audit logic but the MCP registration is opaque, the actual tool definition is built via cognis_core.mcp.build_mcp_server(), making it impossible to verify the full schema from source. This violates the 'see actual tool definition' rule, capping the score at 50 for that tool. The server is STDIO-only, which is a hard transport cap of 50 for protocolReadiness.
Audit a span file or a directory of span files in one shot
Tool definition inferred, not directly visible in source. MCP registration delegated to cognis_core.mcp.build_mcp_server(), cannot verify actual schema, error handling, or output structure.
Tool description is 58 characters, below the 10 - 1024 baseline and lacks WHAT/WHEN/WHY context. Does not explain when to call scan vs other audit tools, what the output contains, or how to interpret findings.
No output schema documented. LLMs cannot plan downstream actions or extract findings structure. The audit() function in JavaScript returns {findings, metrics, failing} but this is not exposed in the MCP tool contract.
Parameter 'target' description is generic ('span file or directory of *.json/*.jsonl'). Does not specify format constraints, path resolution rules, or error cases (e.g., what if file does not exist?).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 38 | 2026-07-28+ | v2 |
No error handling guidance. Tool can fail (invalid JSON, missing spans, broken traces) but LLM has no recovery path. Error responses should guide: 'Try validating JSON with jq' or 'Ensure all spans have span_id'.