Claude-powered training content drift detection system that analyzes training documents for claims that have drifted from live documentation
DocMonitor has three well-named, read-only tools with clear descriptions (avg 180 chars, within baseline 194). All tools have input schemas with type and description. However, output schemas are not formally documented in the code, return types are inferred from implementation (dict/list). Parameter descriptions are present but minimal (e.g., 'days' lacks range/constraint guidance). No error handling guidance, no tool annotations (readOnlyHint present in metadata but not declared), and no pagination support for potentially large result sets. The resource endpoint is well-designed but adds no tool-level complexity.
Analyze training-content markdown for drift against current Claude documentation. Extracts capability claims, fetches the configured doc sources, and classifies each claim as CURRENT, POTENTIALLY_STALE, OUTDATED, or UNVERIFIABLE with cited evidence from the live docs.
Return changelog entries from the last ``days`` days (default 30). Use this to identify what has changed recently in Claude's capabilities so training materials can be checked for newly-deprecated or updated features.
Keyword-search configured Claude documentation for the given query. Returns a ranked list of snippets with source URLs. Use this to verify specific facts or discover where a topic is documented.
Output schemas not formally documented. Return types are inferred from implementation (dict/list) rather than declared in tool metadata. LLMs cannot reliably predict response structure.
Parameter 'days' in get_changelog lacks range constraints. Description does not specify min/max bounds (e.g., 1 - 365), allowing LLMs to pass invalid values.
No error handling guidance. Tools do not document what happens on failure (e.g., doc fetch timeout, invalid query, missing topic). LLMs cannot recover from errors.
search_docs and get_changelog may return large result sets without pagination. No limit parameter or cursor support documented. Large responses waste tokens and risk context exhaustion.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 61 | <=2025-11-25 | v2 |
Tool annotations (readOnlyHint) are declared in metadata but not exposed in tool definitions. LLMs cannot see that these tools are safe to call without confirmation.