Loading ToolBench
Portable, governed X1 agent skills with synthetic evaluations
Critical gaps across all dimensions. Tool descriptions are minimal (e.g., 'X1 eval: get_x1_guide' is 24 chars, below the 50-char baseline for clarity). Most tools lack parameter descriptions entirely, 'documentId' has a description, but 'category', 'clientId', 'limit' in search tools are marked '(optional)' with no guidance on valid ranges or formats. Input schemas are present but sparse: many tools have empty schemas ({}), and those with parameters lack type constraints (e.g., no enum for 'category', no min/max for 'limit'). Output schemas are completely undocumented, LLMs cannot infer what fields to expect from responses. Error handling is absent: no recovery guidance, no categorization of retryable vs fatal errors. Tool composition shows weak chaining: tools return 'structured_result' but downstream tools don't document what fields they consume. Security: no audit logging visible, no permission gates documented, no scope declarations. The mock server registers tools with annotations (readOnlyHint, destructiveHint, idempotentHint) which is positive, but the actual tool definitions in the codebase lack the rigor needed for production agent use.
X1 eval: find_coordination_threads
X1 eval: get_capital_call_job_state
X1 eval: get_capital_call_source_state
X1 eval: get_client_memory
X1 eval: get_coordination_thread
X1 eval: get_my_action_requests
X1 eval: get_user_capabilities
9 of 15 tools have empty input schemas ({}). LLMs cannot infer parameter constraints or types. Tools like get_x1_guide, get_user_capabilities, get_my_action_requests, get_what_matters_now, list_my_coordination_threads, find_coordination_threads, get_client_memory lack any schema definition.
Tool descriptions are minimal and generic (e.g., 'X1 eval: get_x1_guide' is 24 chars). Rubric baseline is 50-200 chars for LLM clarity. No descriptions explain WHEN to use each tool, WHAT it returns, or HOW it differs from similar tools (e.g., search_documents vs search_my_document_contents).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 43 | 2025-06-18+ | v2 |
X1 eval: get_vault_documents
X1 eval: get_what_matters_now
X1 eval: get_x1_guide
X1 eval: list_my_coordination_threads
X1 eval: request_human_confirmation
X1 eval: search_documents
X1 eval: search_my_document_contents
X1 eval: summarize_coordination_thread
Parameter descriptions are missing or trivial. 'category', 'clientId', 'limit' in search/vault tools are marked '(optional)' with no guidance on valid values, ranges, or formats. Rubric requires 'The project key (2-10 uppercase letters)' level of specificity. No enum constraints for 'category'.
Output schemas are completely undocumented. Tools return 'structuredContent: result' but LLMs have no schema to understand what fields are present, their types, or how to chain results to downstream tools. Rubric requires 100% of A+ tools to have documented return types.
No error handling guidance. Mock server throws 'No fixture result for {name}' but production code provides no recovery hints, categorization (retryable vs fatal), or actionable error messages. Rubric requires 'User not found. Try search_users() with a partial name.' level of guidance.