Loading ToolBench
ToolPipe defines 7 tools with mixed quality. Server management tools (toolpipe_pending_servers, toolpipe_list_servers, toolpipe_approve_server, toolpipe_add_server, toolpipe_remove_server) have clear descriptions and reasonable schemas, but lack parameter-level descriptions for some fields and do not declare destructive/idempotent hints. Demo tools (calculate_statistics, generate_customers) have minimal descriptions (under 100 chars) and lack output schema documentation. No tool declares error recovery guidance. Naming is verb-forward but generic (approve, add, remove lack specificity about scope/consent model). Schemas present but incomplete: toolpipe_add_server has mutually exclusive params (command vs url) documented only in description, not enforced in schema. Output structures inferred from code, not formally documented.
Calculate count/min/max/mean over a list of values.
Generate deterministic customer records with id, name, and revenue.
Manually register a server discovery cannot see, then approve it. Provide command/args/env or url, plus a scope.
Approve a discovered server so ToolPipe proxies it. Scope: once (this call), session (this process), workspace (this project), always (everywhere).
List downstream servers with scope and status (env redacted).
List discovered servers waiting for user approval (env redacted).
Demo tools (calculate_statistics, generate_customers) lack substantive descriptions. 'Calculate count/min/max/mean over a list of values' and 'Generate deterministic customer records with id, name, and revenue' are under 100 chars and do not explain when/why to use them or what the output structure is.
No tool declares output schema. LLMs cannot plan downstream calls or extract fields without knowing what fields are returned. E.g., toolpipe_list_servers returns {servers: [{name, via, env, scope, approved, mounted, tools}]} but this is inferred from code, not documented.
Destructive/write tools lack error recovery guidance. toolpipe_remove_server and toolpipe_add_server modify state but do not document what happens on failure, whether operations are idempotent, or how to undo them. No confirmation step for irreversible operations.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 53 | 2025-06-18+ | v2 |
Revoke a server: unmount it and delete persisted grants. Stored results are kept.
toolpipe_add_server has mutually exclusive parameters (command vs url) documented only in description text, not enforced in schema. LLMs may pass both, causing ambiguous behavior. Schema should use oneOf or explicit validation.
Tool names lack specificity about the consent/approval model. 'toolpipe_approve_server' and 'toolpipe_add_server' do not signal that they manage downstream MCP server proxying. Names like 'proxy_approve_server' or 'consent_grant_server' would be clearer.