Loading ToolBench
Semantic search and structured query over content from pluggable sources
Corpus demonstrates solid tool design with clear naming (verb_noun pattern), comprehensive descriptions (100-300 chars), and well-structured schemas. All 10 tools have explicit input schemas with typed parameters and descriptions. However, output schemas are documented only in prose (not formal JSON Schema), and error handling lacks recovery guidance. The semantic/structured query split (corpus_search vs corpus_query) is well-motivated and clearly distinguished. Index tools follow a consistent sanitization pattern. Missing: explicit error classification, confirmation patterns for destructive ops (none present, all read-only), and formal output schema declarations.
Fetch one indexed document by id: full body content and all metadata. Use to escalate from a ``corpus_search`` snippet to the complete record. Returns: A dict with ``id``, ``content`` (full body), and ``meta``; or ``None`` if no document has that id.
Structured, non-semantic metadata query over the indexed mail/documents. Use for time-ranged or "recent" questions (the last day is ``since='1d'``) and for filtering by label or account. Returns: A dict with ``count`` and ``results`` — metadata rows (id, subject, from_addr, sent_at, label), newest first.
Semantic (vector) search over the indexed mail/documents, by MEANING. Use ONLY for topic/similarity lookups ("find emails about the invoice"). Do NOT use for time-based questions — "today", "recent", "this week", "latest" — those MUST use ``corpus_query`` with ``since`` (semantic search ignores dates and will return old, irrelevant matches). Returns: Ranked matches, each with subject, from_addr, sent_at, label, and a 300-character snippet.
Index totals and per-label document counts. Returns: A dict with ``total`` and ``by_label`` (label -> count).
Messages that need an action from you, ranked by importance then urgency.
Output schemas documented only in prose descriptions, not as formal JSON Schema. LLMs cannot parse 'Returns: A dict with count and results' to extract field types and structure for downstream tool chaining.
No error handling guidance. Tools return None or empty dicts on failure without explaining what went wrong or what the LLM should do next (retry, ask user, try alternative tool).
Index tools (index_action_items, index_by_domain) accept free-form 'domain' and 'importance' strings without enum constraints. LLMs may hallucinate invalid values like 'urgent' or 'personal' instead of the actual valid set.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 68 | 2026-07-28+ | v2 |
Recent sanitized items in a life-area domain.
Time-sensitive items, or items carrying a deadline.
Counts of sanitized items by domain and suggested disposition.
The sanitized summary + priority signal for one message id.
Threads awaiting a reply.
corpus_query accepts both 'after'/'before' (absolute ISO-8601) and 'since' (relative) time parameters. Undocumented interaction: 'since' is ignored when 'after' is given. LLMs may pass both expecting AND logic.
No pagination guidance for large result sets. corpus_query defaults to limit=500; corpus_search defaults to top_k=10. No next_cursor or total_count returned, making it unclear if results are truncated or complete.