Loading ToolBench
Six read-only tools with clear, task-shaped names (find_*, customer_*, job_detail). All tools have descriptions (avg 150 chars, within 10-1024 baseline). Input schemas present with types and descriptions. However, output schemas are undocumented, responses are free-text strings, not structured objects. Error handling returns plain text errors without recovery guidance. Missing tool annotations (readOnlyHint). Descriptions are good but could be more concise for LLM parsing.
Balance, estimates, invoices, and payments for one customer, in one response. Use for any "what do they owe" or billing-history question. Needs a customer id; find_customer supplies one.
Full card for one customer: contacts, locations, 5 recent jobs, external ids. Needs a ServiceTitan customer id; use find_customer if you only have a name or phone.
Find a customer by an externalData key/value pair — another system's id. Requires ST_APP_GUID. Use when you hold an id from a CRM synced into ServiceTitan.
Find a customer by name, phone, or external id — no ServiceTitan id needed. One match returns the full card (contacts, locations, recent jobs); several return a short list to choose from. Email lookup is not supported by ServiceTitan.
List jobs with the next appointment inlined. Dates are ISO (2026-08-01). An invalid status returns the valid list, not an empty result. Not for finding a customer — use find_customer.
Output schemas undocumented. All tools return plain-text strings; LLMs cannot parse structured fields or plan downstream calls. Responses should be JSON objects with typed fields (customer_id, name, phone, locations[], jobs[]).
Error handling returns plain text ('ERROR: <message>') without recovery guidance. Should categorize errors (retryable, user-fixable, fatal) and suggest next steps (e.g., 'Customer not found. Try find_customer() with a partial name.').
Missing tool annotations. All tools are read-only but lack readOnlyHint annotation. This prevents agents from reasoning about safety and retry semantics.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 57 | 2026-07-28+ | v2 |
One job with its appointments and assigned technicians. For several jobs at once — a customer's list, a date range, a status — use find_jobs instead of calling this once per job.
find_jobs accepts optional 'status' parameter but does not document valid enum values. Description says 'invalid status returns the valid list', this is opaque to LLMs. Should list valid statuses explicitly.
find_customer and find_by_external_id accept free-form string queries without format constraints. Should document expected formats (phone: E.164, external_id: alphanumeric, etc.) to guide LLM input.