Static source inference · medium confidence · detected: Logging
Deprecated protocol patterns detected
Summary
This server exhibits significant gaps in definition quality across nearly all tools. While the server is well-structured with a clear tool registry and 24 tools spanning traces, observations, scores, datasets, and sessions, the definitions lack the rigor required for production agentic use. Most tools have no visible descriptions in the source code provided, and parameter schemas are not shown for the majority of tools. Only 6 tools (createDataset, getDataset, createDatasetItem, getDatasetItem, deleteDatasetItem, createDatasetRunItem, getDatasetRun) show explicit schema documentation in the audit data; the remaining 18 tools have descriptions listed as empty or missing from the evidence. Naming is generally verb-led (list*, get*, create*, delete*) which is positive, but without accompanying descriptions and parameter schemas, LLMs cannot determine when to select tools or what valid inputs are. The pattern of missing schemas and descriptions for tools like listTraces, getTrace, deleteTrace, listObservations, getObservation, createScore, listScores, getScore, deleteScore, and all score-config tools suggests either incomplete tool registration or missing schema documentation in the source.
Tools (24)
createDatasetwriteauthsource verified80/100
Create a new dataset for evaluation. Datasets contain items with expected inputs/outputs for testing LLM applications.
createDatasetItemwriteauthsource verified83/100
Create or upsert a dataset item. Use id parameter for idempotent updates.
18 of 24 tools lack visible descriptions in source code. Without descriptions, LLMs cannot determine when to select these tools or understand their purpose. Per pattern:tool-description, every tool must have a non-empty description explaining what it does and when to use it.
18 of 24 tools have no visible input schemas in the provided source code. Without input schemas, LLMs cannot validate parameters, understand constraints, or discover optional fields. Per pattern:constrained-input, tools with known parameter sets must declare them as JSON Schema.
Add non-empty descriptions (50 - 200 characters) to all 18 missing tools. For list tools, describe what they retrieve and when to call them. Example: 'List all traces in the project. Use pagination (limit/offset) to retrieve large result sets. Returns trace metadata without nested observations.'
Document input schemas for all tools using JSON Schema. Include type, description, and constraints for each parameter. Example for listTraces: { limit: { type: 'integer', description: 'Max items to return (1-100)', min: 1, max: 100 }, offset: { type: 'integer', description: 'Offset for pagination', min: 0 } }
Document output schemas for all tools. Describe the structure of returned objects, which fields are always present, which are optional, and which IDs/references are included for chaining. Example: 'Returns { traces: [{ id, timestamp, name, metadata, ...}], total: number }'
Add pagination parameters (limit, offset, page_size) to all list tools. Set reasonable defaults (e.g., limit=20) and max caps (e.g., limit max=100). Document in descriptions: 'Results are paginated. Default limit is 20, max 100. Use offset to navigate.'
Add confirmation parameters to destructive tools. Example for deleteTrace: add a 'confirm' boolean parameter (default false) and require it to be true. Or implement a dry-run that returns what would be deleted without actually deleting.
Document error handling for each tool. Add a 'Error Handling' section to descriptions explaining: (1) What errors this tool can return (e.g., 'trace not found', 'invalid dataset name'), (2) Whether they are retryable, and (3) What the LLM should do next. Example: 'If trace not found, call listTraces() to discover available traces.'
Spec posture evidence
Inferred effective spec: <=2025-11-25.
Relies on Logging (deprecated) - log to stderr or use OpenTelemetry
Score history
Overall score trend
↑ 10 points across a rubric change (v1 → v2)
37/100
Scored
Grade
Overall
Spec posture
Rubric
2026-09-22
F
37
<=2025-11-25
v2
2026-03-09
F
27
-
v1
destructiveauth28/100
getDatasetread onlyauthsource verified77/100
Get a dataset by name including its items and run names.
getDatasetItemread onlyauthsource verified77/100
Get a specific dataset item by ID.
getDatasetRunread onlyauthsource verified80/100
Get a specific dataset run by name including its run items.
No output schemas documented for any tool. LLMs need to know what fields to expect in responses so they can plan downstream tool calls and extract the right data. Per pattern:tool, all tools should document their return types and structure.
No pagination parameters (limit, offset, page, cursor) visible in list tools (listTraces, listObservations, listScores, listDatasets, listDatasetItems, listDatasetRuns, listSessions). Returning unbounded results will exhaust context windows. Per pattern:paginated-result, list operations must support limit and offset/cursor parameters.
Destructive tools (deleteTrace, deleteScore, deleteDatasetItem) lack confirmation or dry-run patterns. Per pattern:confirmation-request, irreversible operations should support a confirm-before-execute step or require explicit acknowledgment to prevent agent errors.
No error recovery guidance documented. Tools lack descriptions explaining what to do if operations fail. Per pattern:recovery-guide, error responses must tell the LLM what to do next and categorize errors as retryable, user-fixable, or fatal.
6 tools have descriptions but lack detailed parameter documentation. For example, createDatasetItem has good overall description but parameter descriptions are sometimes sparse (e.g., 'The input for this dataset item (any JSON value)' is vague, should clarify expected structure).
No documentation of idempotency. createDatasetItem mentions 'id parameter for idempotent updates' but does not clarify what happens if called twice with the same id, or whether retries are safe.
createDatasetItem
Add parameter descriptions for all optional fields. For createDatasetItem, clarify the expected structure of 'input' and 'expectedOutput' (e.g., 'JSON object', 'free text', 'structured metrics'). Specify when sourceObservationId requires sourceTraceId.
Document idempotency and retry safety. For createDatasetItem, state: 'If id is provided, repeated calls with the same id will update (not duplicate) the item. Idempotent, safe to retry.'
Add examples to parameter descriptions for enum/constrained fields. For createDatasetItem status: 'Enum: ACTIVE | ARCHIVED. Defaults to ACTIVE.'
Consider adding batch variants for tools called in loops (e.g., deleteMultipleDatasetItems taking an array of IDs vs single deleteDatasetItem). This reduces token waste and latency for common agent workflows.
Add dependency hints in descriptions. For example, getDatasetRun: 'Requires the dataset name and run name. If you only have a trace ID, call listDatasetRuns() first to find matching runs.'
Document what IDs and references are returned by each tool so downstream tools can use them. Example: 'createDataset returns { id, name, description, ... }. Use the returned id with createDatasetItem.' Ensure chaining is seamless.