Loading ToolBench
Two tools with clear verb-noun naming (search_email, get_email) and reasonable descriptions (194 and 97 chars respectively). Input schemas are present with proper JSON Schema structure and type definitions. However, parameter descriptions lack actionable detail (e.g., 'mu find query string' does not explain syntax or constraints), output schemas are undocumented, and error handling is minimal. No tool annotations (readOnlyHint, etc.). Results are capped at 20 but no pagination mechanism. Descriptions meet baseline length but lack LLM-optimization guidance on when to use each tool vs alternatives.
Fetch the full body and headers of an email by its file path (from search_email results).
Search emails using mu find query syntax. Examples: 'from:alice subject:invoice', 'maildir:/work/INBOX date:7d..now', 'flag:unread AND NOT flag:trashed'. Returns a list of matching messages with metadata.
Output schemas not documented. LLMs cannot infer the structure of search_email results (fields like :path, :subject, :from, :date, :flags, :message_id, :maildir, mu4e_link) or get_email response (subject, from, to, date, message_id, mu4e_link, body). Agents must guess field names for downstream operations.
Parameter descriptions lack actionable constraints. 'mu find query string' does not explain valid syntax, examples of operators (from:, subject:, flag:, date:, maildir:), or error cases. LLMs cannot validate input before calling. Should document: 'Supports operators: from:, subject:, flag:, date:, maildir:, AND, NOT. Example: "from:alice subject:invoice date:7d..now"'.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Both tools are read-only and idempotent, but this is not declared in the schema. Agents cannot infer safety properties without explicit hints.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 67 | 2026-07-28+ | v2 |
Error handling returns generic 'Error: {e}' messages with no recovery guidance. Example: if get_email receives an invalid path, the response is 'Error: [Errno 2] No such file or directory', the LLM cannot infer that it should call search_email first to get valid paths. Should return: 'File not found. Use search_email to find valid email paths.'
No pagination or result limit documentation. search_email caps at 20 results by default, but the description does not state this limit or explain how to retrieve more results. Agents may assume all matching emails are returned and miss important messages.