Loading ToolBench
Smart invoicing service with auto-chase payment reminders
InvoicePilot has 6 tools with clear verb-noun naming (create_invoice, list_invoices, etc.) and reasonable descriptions (avg ~120 chars). All tools have input schemas with typed parameters. However, descriptions lack actionable context for LLM selection, they state WHAT but not WHEN or WHY. Parameter descriptions are minimal (e.g., 'The ID of the invoice' without format/constraint hints). Output schemas are undocumented; tools return free-text strings rather than structured JSON. Error handling is absent, no guidance on retries, user-fixable errors, or recovery paths. Missing tool annotations (readOnlyHint, destructiveHint). Composition is sound: tools are single-responsibility and chainable via invoice_id.
Create a new invoice. Items should be list of {description, qty, rate}. Set send_immediately=True to email the invoice to the client right away.
Show all unpaid invoices with amounts and days overdue.
Get revenue summary: total revenue, outstanding amounts, overdue amounts, this month stats.
List all invoices, optionally filtered by status (draft/sent/overdue/paid).
Mark an invoice as paid.
Manually trigger a payment reminder for an overdue invoice.
Output schemas undocumented. All tools return free-text strings; LLMs cannot parse structured data or chain results. E.g., list_invoices returns a formatted string instead of {invoices: [{id, number, client_name, total, status, due_date}], count, total_outstanding}.
Descriptions lack WHEN/WHY context. E.g., send_reminder says 'Manually trigger a payment reminder' but doesn't explain when to use it vs. automatic reminders, or what happens if the invoice is already paid. LLMs cannot distinguish between similar tools.
No error handling guidance. Tools return bare error strings ('Error: Invoice #X not found') without recovery hints. LLMs don't know whether to retry, ask the user, or call a discovery tool.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 60 | <=2025-11-25 | v2 |
Missing tool annotations. No readOnlyHint on list_invoices/get_outstanding/get_revenue_summary, no destructiveHint on create_invoice/send_reminder/mark_paid. Agents cannot infer safety/idempotency.
Parameter descriptions are minimal. E.g., 'The ID of the invoice to mark as paid' lacks format/type hints. 'status' in list_invoices says 'Optional filter' but doesn't enumerate valid values (draft/sent/overdue/paid) as an enum constraint.