Loading ToolBench
Server has 4 tools with basic schemas and descriptions, but significant quality gaps. Tool names lack clear action verbs (plot_vis_columns, hdf5_file_listing are vague). Descriptions are present but generic (10-50 chars, below 194-char baseline). Parameter descriptions are minimal or missing context. No output schemas documented. Error handling is present in handlers but not surfaced to LLM. No tool annotations (readOnlyHint, destructiveHint). slurm_job_submission is a destructive tool with no confirmation pattern. Overall structure is functional but falls short of production-grade agent tool standards.
List HDF5 files in directory
Plots two columns from a CSV file. Default to first two columns if not specified.
Report number of CPU available on the system
Submit jobs to Slurm scheduler
Tool names lack action verbs. 'plot_vis_columns' should be 'plot_csv_columns'; 'hdf5_file_listing' should be 'list_hdf5_files'. LLMs infer intent from verb-noun structure.
Descriptions are too short (10-50 chars vs 194-char baseline). 'Report number of CPU available on the system' lacks context on when/why to call it. Descriptions should explain WHAT, WHEN, and WHAT IT RETURNS.
No output schemas documented. Handlers return structured dicts (e.g. {file, metadata}) but LLM has no schema to plan downstream calls. Document return types for all tools.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 39 | 2026-07-28+ | v2 |
slurm_job_submission is destructive (submits jobs to scheduler) but has no confirmation pattern, dry-run option, or destructiveHint annotation. Agents can accidentally submit hundreds of jobs.
Parameter descriptions are minimal. 'column x' and 'column y' lack format guidance. 'path_pattern' in hdf5_file_listing does not explain glob syntax or expected format. Descriptions should state constraints and format.