Loading ToolBench
Goal-driven agent runtime with tool discovery, credential management, and MCP server integration
EngineX exhibits significant definition quality gaps across 24 tools. While all tools have descriptions (10-200 chars, mostly adequate), input schemas are present but inconsistently detailed. Many tools lack parameter type constraints (enums, ranges, patterns). Critical issues: (1) Tools like 'load_skill', 'categorize_issue', 'web_search' have minimal descriptions (20-80 chars) below the 50-200 char LLM-optimized baseline. (2) Parameters frequently lack constraints, 'query' in web_search has no length/format guidance; 'severity' in notify_slack/notify_pagerduty lacks enum values (should be: critical|high|medium|low). (3) Output schemas are not documented, LLMs cannot infer what fields to expect from tools like 'fetch_broker_transactions' or 'run_log_monitor_pipeline'. (4) Error handling is absent, no recovery guidance, no actionable error messages. (5) Tools like 'chart_render' accept 'spec' as generic object without schema documentation. (6) No tool declares idempotency, permissions, or scope. Average per-tool score: 42/100.
Append text to a file in the agent data directory.
Parse incidents JSON and return dispatch plan with alert targets.
Categorize the customer feedback into predefined buckets.
Render a chart or diagram to a high-DPI PNG and return a rich envelope so the chat can also render the chart live.
Correct transaction discrepancies.
Create a Jira ticket for an incident.
Fetch broker transactions from last hour.
Output schemas undocumented. Tools like 'fetch_broker_transactions', 'run_log_monitor_pipeline', 'normalize_transactions' have no documented return types. LLMs cannot infer what fields to extract or pass to downstream tools.
Enum constraints missing for categorical parameters. 'severity' in notify_slack, notify_pagerduty, create_incident_ticket lacks enum values (should be: critical|high|medium|low). 'kind' in chart_render should be enum: echarts|mermaid. LLMs will hallucinate invalid values.
No error handling guidance. Tools provide no recovery hints, error classification, or actionable messages. E.g., 'web_search' and 'web_scrape' do not document what happens on network failure, rate limit, or invalid URL. Agents cannot self-correct.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 47 | 2026-07-28+ | v2 |
Fetch investor logs from last hour.
List files in the agent data directory.
Load a slice of a file from the agent data directory.
Load a skill SKILL.md body by name for guidance during this task.
Normalize raw transaction data.
Trigger PagerDuty for SEVERE incidents.
Send a Slack alert for an incident.
Validate production config and integration health before monitoring.
Record learning outcome for fingerprint tuning and mute window.
Reset correction attempt state for a new run.
Run one monitoring tick: Grafana fetch, dedup, and rule-based scoring.
Save text content to a file in the agent data directory.
Return a file URI for the user (optionally open in browser).
Store verified transaction results.
Validate transaction reconciliation.
Fetch a URL and return extracted page text.
Search the web for a query (Brave API when configured, else demo results).
No idempotency or destructive operation warnings. Tools like 'correct_transactions', 'store_verified_results', 'notify_slack', 'notify_pagerduty' modify state but lack idempotentHint or destructiveHint annotations. Agents cannot reason about retry safety.
Parameter constraints missing. 'count' in web_search lacks min/max (should be 1-10). 'max_chars' in web_scrape lacks bounds. 'offset_bytes' and 'limit_bytes' in load_data lack ranges. 'width', 'height', 'dpi' in chart_render lack constraints. Unbounded parameters invite invalid LLM inputs.