Loading ToolBench
Server has 29 tools with consistent naming (verb_noun pattern) and complete input schemas. All tools have descriptions (10-50 chars, mostly terse). Schemas are well-formed with proper types, enums, and constraints. However, descriptions are uniformly brief and lack context for LLM selection, they state WHAT but not WHEN or WHY. Parameter descriptions are minimal or absent. Output schemas are not documented. Error handling is generic (ToolError wrapper). No tool annotations (readOnlyHint/destructiveHint). Risk labels are present but not exposed to the LLM.
Create a dataset. Destructive: does not check for existing data.
Create an NFS share.
Create an SMB share.
Create a ZFS snapshot.
Delete a dataset. Destructive. Runs as a TrueNAS job.
Delete an NFS share by id.
Delete an SMB share by id.
Tool descriptions are uniformly terse (10-50 chars) and lack context. They state WHAT (e.g. 'List NFS shares') but not WHEN to use or how they differ from similar tools. LLMs cannot reliably select between list_nfs_shares, list_smb_shares, list_iscsi_targets without richer descriptions.
Parameter descriptions are missing or minimal. QueryFilter.field, QueryFilter.op, and QueryFilter.value lack descriptions explaining what fields are queryable, what operators mean, or what value types are expected. This forces LLMs to guess.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 49 | <=2025-11-25 | v2 |
Delete a ZFS snapshot by id (e.g. 'tank/foo@snap1').
Get a dataset by path.
Return the global network configuration.
Get a pool by id (int) or name (str).
Return TrueNAS version, hostname, uptime, hardware summary.
Return the current update status.
List certificates.
List datasets, optionally scoped to a pool.
List physical disks.
List iSCSI targets.
List NFS shares.
List NVMe-oF subsystems.
List ZFS pools.
List replication tasks.
List services (use filters to narrow, e.g. service = 'ssh').
List SMB shares.
List snapshots, optionally scoped to a dataset.
List local users.
Restart a TrueNAS service. Runs as a job.
Run a replication task by id. Runs as a TrueNAS job.
Start a TrueNAS service (e.g. 'ssh', 'smb'). Runs as a job.
Stop a TrueNAS service. Runs as a job.
Output schemas are not documented. Tools return results but the response structure (fields, types, pagination) is not visible in the tool definition. LLMs cannot plan downstream calls or extract required IDs without seeing what fields are returned.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Risk labels exist in metadata but are not exposed via MCP tool annotations. LLMs cannot distinguish safe reads from destructive operations without explicit hints.
Error handling is generic. _to_tool_error() wraps all TrueNAS errors as ToolError with minimal context. Errors do not guide recovery (e.g. 'Not found: X' does not suggest 'Try list_pools() first'). LLMs cannot self-correct.