Loading ToolBench
A persistent, agent-first compute broker with shared CLI and MCP interfaces
Server has 14 well-named tools with clear action verbs (pool_overview, list_workers, submit_job, etc.). All tools have descriptions (10 - 200 chars, within baseline range). Input schemas are visible and typed for most tools. However, output schemas are NOT documented, LLMs cannot predict return structure. Parameter descriptions are present but minimal (10 - 30 chars); many lack constraint details (e.g., 'limit' has no min/max). Error handling is absent from tool definitions. Security: submit_job accepts JobSpec (complex object) but no validation guidance. Composition is strong, tools chain well (submit_job → get_job → list_artifacts → fetch_artifact). No tool combines multiple concerns. Naming follows verb_noun convention consistently.
Request cancellation; status changes only when confirmed by the worker.
Verify artifact hash and return its local path; content requires explicit true and is bounded.
Recover durable job state across agent sessions and broker restarts.
Bounded logs; offset counts backwards from the end of the available log window.
Return runs and recorded conclusions for a project.
Return canonical paths, sizes, and hashes before reading content.
List persistent jobs with pagination.
Output schemas not documented. LLMs cannot predict return structure (dict vs list, field names, types). Breaks downstream tool chaining and forces exploratory calls.
Numeric parameters (limit, offset, tail, max_bytes) lack min/max constraints in descriptions. LLMs may pass invalid values (negative, zero, unbounded).
No error handling guidance in tool definitions. Agents cannot distinguish retryable errors (network timeout) from user-fixable (invalid job_id) from fatal (permission denied).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 64 | 2026-07-28+ | v2 |
Discover finalized results from earlier sessions; check status before interpreting.
List normalized worker capabilities without credential profiles.
Show worker capacities separately and durable job counts.
Attach scientific interpretation separately from immutable infrastructure artifacts.
Restart exhausted artifact retrieval only; never rerun the compute job.
Persist an experiment and return its job ID quickly; existing project policy controls dispatch.
Refresh authoritative provider quotas; unknown is not zero.
submit_job accepts JobSpec (complex nested object) with no validation rules in description. LLMs cannot infer required fields, valid enum values for 'phase', or constraints on 'resources'.
Pagination tools (list_jobs, list_ready_results, get_project_runs) return lists but do not document total count or next_cursor. Agents cannot determine if more results exist.