Loading ToolBench
MiniTask has clear, action-oriented tool names (list_, create_, update_, complete_, set_) and reasonable descriptions (avg ~100 chars). All 7 tools have input schemas with type definitions. However, parameter descriptions are sparse or missing context. For example, 'date' parameters lack format validation hints; 'uri' parameters lack guidance on where to obtain them. Output schemas are not documented. Error handling is minimal, no recovery guidance or actionable error messages. The server lacks tool annotations (readOnlyHint, destructiveHint) despite having clear risk profiles (READ_ONLY, WRITE, DESTRUCTIVE). Composition is sound: tools are single-purpose and chainable (list_tasks → update_task via uri). Overall, definitions are functional but lack LLM-optimization depth.
Set all overdue tasks to today. Same as 'Catch Up' in the desktop app.
Mark a task as done — removes it from the server, same as checking it off in the app.
Create a new task. Date format: YYYY-MM-DD (e.g. 2026-05-30). Defaults to today if omitted.
List all available calendars on the CalDAV server.
List all open (incomplete) tasks in the current calendar.
Switch to a different calendar by URL. Use list_calendars to find the URL.
Missing output schema documentation. Tools return structured data (dicts, lists) but LLMs cannot see what fields to expect. E.g., list_calendars returns list[dict] with unknown structure; list_tasks returns dicts with 'id', 'uri', 'title', 'date', 'starred', 'completed', 'note' but this is not documented in the tool definition.
Parameter descriptions lack actionable constraints. 'date' parameter in create_task and update_task states format 'YYYY-MM-DD' in the tool description but not in the parameter description itself. LLMs cannot read tool docstrings, they parse parameter descriptions. Missing: min/max length, regex pattern, or explicit format constraint in the schema.
No tool annotations despite clear risk profiles. Tools marked DESTRUCTIVE (complete_task) or WRITE (create_task, update_task, set_calendar, catchup) lack readOnlyHint, destructiveHint, or idempotentHint annotations. LLMs cannot distinguish safe from risky tools without explicit hints.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 61 | 2026-07-28+ | v2 |
Update an existing task. The URI comes from list_tasks. Only pass the fields you want to change; omitted fields keep their current value.
Minimal error handling and no recovery guidance. Code raises RuntimeError('MiniTask is not configured yet...') but does not guide the LLM on next steps. No categorization of errors as retryable, user-fixable, or fatal. No actionable error messages for invalid inputs (e.g., malformed date, invalid calendar_url).
'uri' parameter in update_task and complete_task lacks guidance on how to obtain it. Description says 'Task URI (from list_tasks)' but does not explain the field name or structure. LLMs may struggle to extract the correct field from list_tasks output.