Loading ToolBench
AI agent runtime with schema-validated recorded plans and bounded tool execution. Supports deterministic plan generation, tool execution with retry logic, memory persistence, and MCP server integration.
APEX provides 7 tools with basic schemas and descriptions, but falls short of production quality. Tool names are action-oriented (shell, read_file, write_file, http_get, rag_multi_query, memory_read, memory_write), which is good. However, descriptions are minimal (10-50 chars for most tools), parameter descriptions are sparse or missing, and output schemas are incompletely documented. The code shows explicit Tool definitions with input_spec and output_spec, but parameter-level descriptions are absent from the schema objects. Error handling is minimal, tools raise exceptions without recovery guidance. No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear risk profiles (shell is DESTRUCTIVE, write_file is WRITE). The rag_multi_query tool has the most detailed description but still lacks parameter constraints (hops range 1-5 is enforced in code, not declared in schema). Overall, definitions are functional but lack the LLM-optimization and completeness expected of production tools.
Perform an HTTP GET request with optional headers and return response body and status code
Read from persistent key-value memory store. Returns a single entry by key or all entries if no key specified
Write to persistent key-value memory store, creating or updating entries
Multi-hop RAG (Retrieval-Augmented Generation) query with configurable hops, returning answer, hop summaries, and sources
Read the contents of a file as UTF-8 text
Execute shell commands with captured stdout, stderr, and exit code
Write content to a file, creating parent directories as needed
Parameter descriptions missing or minimal. Input schemas define types (str, dict, int) but lack descriptions explaining what each parameter controls, valid ranges, or constraints. E.g., 'cmd' in shell has no description; 'hops' in rag_multi_query enforces 1-5 in code but schema declares no bounds.
Tool descriptions are too brief (10-50 chars). Rubric baseline is 194 chars average; APEX tools average ~35 chars. Descriptions lack WHEN to use, WHAT it returns, and prerequisites. E.g., 'Execute shell commands...' does not explain risks, output format, or when to prefer this over other tools.
No tool annotations despite clear risk profiles. shell is DESTRUCTIVE, write_file is WRITE, read_file is READ_ONLY. MCP spec supports readOnlyHint, destructiveHint, idempotentHint annotations, none are present. LLMs cannot infer safety properties without explicit hints.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 55 | 2026-07-28+ | v2 |
Output schemas incomplete. Tools define output_spec as dicts (e.g., {'stdout': str, 'stderr': str, 'code': int}) but do not document field meanings, ranges, or when fields may be absent. rag_multi_query returns 'hop_summaries' (list) and 'sources' (list) with no description of structure or content.
Error handling lacks recovery guidance. Tools raise RuntimeError or generic exceptions (e.g., 'RAG request failed at hop X') without actionable next steps. Rubric requires: 'User not found. Try search_users().' APEX provides no such guidance, forcing LLMs to guess recovery strategies.