Concierge: agentic customer-support drafting with a deterministic policy gate. Sanitized public sample.
Strong foundation with complete schemas and clear descriptions. All 7 tools have explicit input schemas with proper Zod validation, type definitions, and parameter descriptions. Tool names follow verb_noun convention (mail.read, mail.search, shop.orderStatus). Descriptions are concise (10-60 chars) and action-oriented. Output types are well-defined interfaces. However, descriptions are at the lower end of the 10-1024 char baseline (avg ~35 chars here vs 194 baseline), lacking context on WHEN to use each tool or dependencies between them. No error handling guidance, no recovery paths, and no output schema documentation in the tool registry itself. Security model is sound (no send path for mail, synthetic data only), but lacks explicit permission declarations and audit trail guidance.
Search the internal knowledge base for policy and product answers.
Stage a reply draft for human review. No send path.
Read a full mail thread by id.
Search mail threads by query.
Look up a customer profile and prior contact by email.
Look up live stock level by sku.
Look up authoritative order state by order id.
Tool descriptions lack context on WHEN to use each tool and dependencies. E.g., mail.draft says 'Stage a reply draft' but doesn't explain it requires a prior mail.read or mail.search call, or that it's the only write path.
No output schema documentation in the tool registry. While TypeScript interfaces exist (MailThread, OrderStatus, etc.), the TOOL_REGISTRY entries lack 'output' field declarations. LLMs cannot see what fields to expect without reading source code.
No error handling or recovery guidance. Tools return synthetic data only; no error cases documented. If a threadId doesn't exist, what happens? If an email is invalid, what's the recovery path?
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 57 | 2026-07-28+ | v2 |
Parameter descriptions are minimal (10-30 chars). E.g., 'Thread identifier, minimum 1 character' lacks context on format (UUID? alphanumeric?). Baseline is 72 chars avg. Expand to include format hints and examples of valid values.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). The 'access' field in ToolDescriptor is custom and not part of MCP spec. Should use standard tool annotations to signal read-only vs draft operations.