A synthetic, production-shaped service-business agent built to demonstrate forward-deployed AI engineering.
Three tools with basic schemas and descriptions, but significant gaps in parameter documentation and output structure. All tools have descriptions (10-60 chars), but descriptions are minimal and lack context for LLM selection. Input schemas present but parameters lack detailed descriptions explaining constraints, formats, or dependencies. No output schemas documented. Error handling is absent, tools return success responses without guidance on failure modes or recovery. No tool annotations (readOnlyHint/destructiveHint). Naming is clear (verb_noun pattern), but descriptions do not explain WHEN to use each tool or what happens on failure.
Collect scheduling details without claiming that an appointment is confirmed.
Queue a request for a human staff member.
Search approved service-business knowledge and return citations.
Parameter descriptions missing or minimal. 'name', 'callback_number', 'service', 'preferred_time' in collect_intake lack format constraints, validation rules, or examples. LLMs cannot infer whether 'callback_number' expects E.164 format, whether 'service' is free-text or enum, or what 'preferred_time' format is (ISO 8601, natural language, time window?).
No output schemas documented. Tools return dict[str, object] or dict[str, str] with no specification of fields, types, or structure. LLMs cannot plan downstream calls or extract required data. E.g., search_service_knowledge returns response.model_dump() but no schema is visible for what fields are in that response.
No error handling or recovery guidance. Tools return success responses only. No indication of failure modes, retryability, or what the LLM should do if a call fails. E.g., collect_intake always returns 'pending_confirmation', what if the callback_number is invalid? What if the service type is not recognized?
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 54 | 2026-07-28+ | v2 |
Tool descriptions are too brief (35-40 chars) and lack context. 'Collect scheduling details without claiming that an appointment is confirmed' is clear but does not explain WHEN to use this vs escalate_to_human, what happens after collection, or whether the tool validates inputs. Descriptions should be 50-200 chars with WHAT, WHEN, and WHAT-HAPPENS guidance.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). collect_intake and escalate_to_human are WRITE operations but lack destructiveHint annotation. LLMs cannot determine which tools are safe to retry or which have irreversible side effects.