Loading ToolBench
Benchmarks LLMs via remote SSH connection to KoboldCPP. Analyzes model metadata, detects system capabilities, and runs benchmarks with configurable parameters.
Server has 4 tools with basic descriptions but critical gaps in schema completeness, parameter documentation, and error handling. Only 1 of 4 tools has a visible input schema (analyze_model and run_benchmark_via_ssh have partial schemas in the prompt, but setup_ssh_connection and detect_system_capabilities lack any documented input schema). Descriptions are present but generic (10-60 chars), below the 50-200 char LLM-optimized baseline. No output schemas documented. No error recovery guidance. No parameter constraints (enums, ranges). Tool names are reasonable (verb_noun pattern) but descriptions fail to explain WHEN to use each tool or what prerequisites exist.
Extract model metadata using GGUF reader on remote system via SSH
Discover remote system VRAM and capabilities via SSH
Execute benchmark command via SSH and parse results
Establish SSH connection to remote system
setup_ssh_connection and detect_system_capabilities have NO input schemas visible.
Descriptions are too short (10-60 chars) and lack WHEN/WHY context. 'Establish SSH connection to remote system' does not explain prerequisites, dependencies, or when to call it vs other tools. Baseline is 50-200 chars with actionable context.
No output schemas documented. LLMs cannot plan downstream calls or extract required fields. E.g., what does setup_ssh_connection return? What fields does analyze_model return beyond the visible code snippet?
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 38 | 2026-07-28+ | v2 |
run_benchmark_via_ssh accepts 'other_args' as free-form array with no validation, enum constraints, or format guidance. LLMs will hallucinate invalid command-line arguments. Should document allowed flags or use constrained enum.
No error handling guidance. Code raises RuntimeError with messages like 'SSH connection not established' and 'Remote GGUF analysis failed' but does not tell LLM what to do next (retry? call setup_ssh_connection first? ask user?). Per pattern:recovery-guide, errors must guide recovery.