Loading ToolBench
A Trace-Driven, Role-Aware Benchmark for Agent Operations in Realistic (HPC) Environments
AOBench defines 5 tools with schemas and descriptions, but quality is inconsistent. All tools have descriptions (10-50 chars, mostly adequate), and input schemas are present with parameter types. However, parameter descriptions are sparse or generic ('Filter jobs by user (optional, for query_jobs)'), and output schemas are completely undocumented. No tool declares what it returns or how results chain to downstream tools. Error handling is absent, no guidance on retryability, user-fixable vs fatal errors, or recovery paths. Tool names are generic verbs (slurm, docs, rbac, telemetry, facility) that do not clearly convey action intent; 'slurm' and 'facility' are nouns, not verb_noun patterns. Parameters like 'method' are overloaded, each tool accepts a 'method' string that dispatches to different sub-operations, violating single-responsibility and forcing LLMs to reason about method dispatch rather than tool selection. No security annotations, no idempotency hints, no composition guidance.
Mock docs tool — keyword search over environment documentation files
Mock facility tool — reads power, rack, and inventory data from environment bundle
Mock RBAC tool — checks permissions from environment policy file
Mock SLURM tool — reads deterministic scheduler state from environment bundle
Mock telemetry tool — reads CSV/Parquet telemetry from environment bundle
Tool names are nouns or generic verbs, not action-oriented verb_noun patterns. 'slurm', 'docs', 'rbac', 'facility' do not convey what action the tool performs. LLMs cannot infer intent from the name alone.
All tools use a 'method' parameter that dispatches to multiple sub-operations (e.g., slurm.method in [query_jobs, job_details, list_nodes, list_partitions]). This violates single-responsibility and forces LLMs to reason about method dispatch rather than selecting the right tool. Should split into separate tools: query_jobs, get_job_details, list_nodes, list_partitions.
Output schemas are completely undocumented. No tool declares what fields it returns, what types they are, or how results chain to downstream tools. LLMs cannot plan multi-step workflows or extract required IDs for follow-up calls.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 48 | 2026-07-28+ | v2 |
Parameter descriptions are generic or missing context. 'Filter jobs by user (optional, for query_jobs)' does not explain format (username? email? ID?), constraints, or when to use it. No descriptions for 'method' parameter itself, forcing LLMs to infer valid values.
No error handling guidance. Tools do not document what errors are possible, whether they are retryable, or what the LLM should do next. A failed query returns nothing, no recovery path.