Loading ToolBench
Local-first agent communication server with HTTP control plane and MCP transport
Conclave MCP exposes 6 tools across HTTP endpoints and CLI commands. Critical gaps: (1) Most tools lack explicit input schemas in the source code, only 2 of 6 have visible parameter definitions with types. (2) Tool names are HTTP verbs (GET, POST) or CLI subcommands (admin vacuum, audit list) rather than action-verb naming (e.g., 'check_health', 'stream_room_events', 'create_api_key'). (3) Descriptions exist but are brief (50 - 150 chars) and lack LLM-optimized guidance on WHEN to use each tool or what happens next. (4) No documented output schemas, responses are inferred from context. (5) Error handling is not visible in the provided code; no recovery guidance. (6) Security-sensitive tools (admin vacuum, admin keys create-bootstrap) lack permission/scope documentation.
Server-Sent Events (SSE) stream of new messages in a room with heartbeat and configurable timeout
JSON-RPC-over-HTTP proxy that forwards requests to the Conclave control plane via MCP server
First-use setup command that creates the first API key directly in the SQLite file. Must be run with server stopped. Returns plaintext token exactly once.
Offline maintenance command that runs VACUUM on the SQLite file to reclaim space. Must be run with server stopped.
Query the audit log with filtering by actor, action, room, and timestamp. Requires admin-scoped API key.
Health check endpoint that pings the database and returns server status
Tool names do not follow verb_noun convention. HTTP verbs (GET, POST) and CLI subcommands (admin vacuum, audit list) are not action-oriented. LLMs cannot infer intent from 'POST /v1/mcp' or 'admin vacuum'. Rename to 'stream_room_events', 'proxy_mcp_request', 'reclaim_database_space', 'create_api_key', 'query_audit_log'.
Input schemas are not visible or incomplete for 4 of 6 tools. Only 'GET /v1/rooms/{id}/events' and 'audit list' show parameter definitions with types. 'healthz', 'POST /v1/mcp', and 'admin vacuum' lack explicit schema documentation in the source.
No documented output schemas. Responses are inferred from context (e.g., 'returns server status', 'returns plaintext token'). LLMs cannot plan downstream tool calls or extract required fields without explicit response schemas. Document what each tool returns: field names, types, and whether results are paginated.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 39 | 2026-07-28+ | v2 |
Security-sensitive tools ('admin vacuum', 'admin keys create-bootstrap') lack permission/scope declarations. No documentation of who can call these tools, what credentials are required, or what audit trail is created. Per pattern:scope-declaration, each tool should declare required permissions (e.g., 'admin:database', 'admin:keys').
Descriptions are generic and lack LLM-optimized guidance. 'Health check endpoint that pings the database' (60 chars) does not explain WHEN to call it or what to do if it fails. 'JSON-RPC-over-HTTP proxy' (30 chars) is too brief. Expand to 50 - 200 chars with context: 'Check server health before making requests. If this fails, the server is down or database is unreachable. Returns status and latency.'