Loading ToolBench
Governed AI-ops for managed-endpoint fleets (thin clients / VDI): login-storm & drift analysis with a built-in governance harness (audit, budget, undo, risk tiers)
11 tools with complete input schemas and descriptions. Naming follows verb_noun pattern (drift_report, endpoint_list, session_storm). Descriptions are adequate (50-150 chars) but lack LLM-optimized guidance on WHEN to use each tool vs. similar ones. Parameters are typed and described, but lack constraints (enums, ranges). Output schemas are not documented, LLMs cannot plan downstream calls. Risk tiers (READ_ONLY, REVERSIBLE, IRREVERSIBLE) are declared but not exposed in tool definitions. Error handling and recovery guidance are absent from descriptions.
Patch-level distribution + which endpoints are behind the target.
Report endpoints drifted from the fleet-majority baseline.
Assign a config profile to an endpoint (reversible; dry-run + confirm).
Show one managed endpoint by id.
List all managed endpoints.
Request an endpoint reboot (no undo; dry-run + confirm).
One-shot fleet health: online/offline, stale endpoints, version spread.
Output schemas not documented. LLMs cannot infer what fields each tool returns, forcing them to guess downstream parameter mappings. E.g., does drift_report return endpoint_id or endpoint_name? Does fleet_overview include version_spread as a string or structured object?
Descriptions lack WHEN-to-use guidance. 'List all managed endpoints' does not explain when to call endpoint_list vs. fleet_overview, or whether endpoint_list is paginated. LLMs need comparative context to select the right tool.
Numeric parameters lack range constraints. session_storm accepts since_hours (1..720) and window_s (300 default) but descriptions do not state min/max. LLMs may pass invalid values (0, -1, 999999) without validation guidance.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 60 | <=2025-11-25 | v2 |
List recent login/boot sessions.
Detect login storms and rank the slowest login/boot contributors.
Apply a recorded undo (dispatches its inverse tool; dry-run + confirm).
List recorded, not-yet-applied undo tokens.
Risk tiers (READ_ONLY, REVERSIBLE, IRREVERSIBLE) are declared in metadata but not exposed as tool annotations in the MCP schema. LLMs cannot see which tools are safe to retry vs. destructive without reading descriptions.
Error handling and recovery guidance absent. Descriptions do not explain what to do if target is not found, or if an endpoint is offline. E.g., 'If target not found, call fleet_overview to list available targets.'