MCP server exposing portfolio tools (Semantic Search, Eval Framework, Observability)
Server has 5 tools with complete schemas and descriptions. Naming follows verb_noun convention (search_, run_, list_, get_). Descriptions are present (avg 150 chars) and explain WHAT the tool does. However, parameter descriptions lack actionable constraints (no ranges, patterns, or format guidance). Output schemas are not documented, LLMs cannot predict response structure. Error handling is minimal; no recovery guidance. Tool composition is sound (each does one thing), but missing idempotency hints and chaining IDs in responses.
Get a specific prompt by name with full metadata including version, score history, and template content.
List available prompts from the Prompt Registry. Supports pagination and optional filtering by tag.
Run an evaluation suite from the Eval Framework. Executes test cases against a prompt variant and returns scores and pass/fail results.
Search documents using semantic similarity. Connects to the Semantic Search Engine to find relevant documents based on a natural language query.
Check health of all portfolio services: Semantic Search API and Eval Framework. Returns status, latency, and error details for each component.
Output schemas not documented. LLMs cannot predict response structure (fields, types, nested objects). Forces agents to guess what data is available after each call.
Parameter descriptions lack actionable constraints. 'limit' has min/max but no guidance on typical values or performance impact. 'query' has no format hints (length, special chars). 'suite' and 'variant' have no enum or discovery guidance.
No error handling guidance. Tools do not document what errors are possible, whether they are retryable, or what the LLM should do next. E.g., toad_search_documents could fail if the Semantic Search API is down, no recovery hint.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 55 | 2026-07-28+ | v2 |
toad_run_eval is marked WRITE but has no confirmation or dry-run capability. Agents could accidentally run expensive eval suites. No idempotency hint, unclear if retrying the same call is safe.
Response chaining IDs missing. If toad_list_prompts returns prompts, does each include a 'name' field that toad_get_prompt accepts? If toad_search_documents returns documents, do they include IDs for downstream tools? Undocumented breaks tool chains.