Loading ToolBench
Run untrusted MCP servers in isolated cages with network filtering and secret management
Tool definitions are present with basic structure but lack depth. All 7 tools have names and descriptions, but descriptions are brief (avg ~80 chars, below the 194-char baseline). Input schemas are visible and typed, but parameter descriptions are minimal or missing context. No output schemas documented. Error handling guidance absent. Tools like 'save_note' and 'latest_skill' have vague descriptions that don't explain when to use them or what they return. Parameter descriptions exist but are often generic (e.g., 'A city or place name'). No tool annotations (readOnlyHint/destructiveHint) despite clear risk classifications. Composition is reasonable (single responsibility per tool), but chaining support is unclear due to missing output documentation.
Look up the coordinates of a place name, and disambiguate between places that share one. Useful when a forecast came back for the wrong Springfield.
Get the weather happening right now at a place.
Get a daily weather forecast for a place.
Return the latest mcpvessel skill for an MCP client, fetched live from the repository, so an agent can keep its installed skill current without upgrading the mcpvessel binary. The skill (in this repo at internal/clientskill/skills/<client_id>/SKILL.md) evolves faster than the binary; this serves whatever is on the default branch right now.
Save a short note and return a confirmation.
Search mcpvessel's source on GitHub for how something is implemented, when the docs do not answer. Needs the GITHUB_PERSONAL_ACCESS_TOKEN secret because GitHub code search requires authentication.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields without knowing response structure.
Tool descriptions are too brief (avg ~80 chars vs 194-char baseline). 'save_note' (28 chars), 'search_code' (145 chars), 'latest_skill' (180 chars) lack context on when to use them or what they return.
No tool annotations (readOnlyHint/destructiveHint/idempotentHint) despite clear risk classifications. 'save_note' is WRITE but unmarked; weather tools are READ_ONLY but unmarked.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 56 | 2026-07-28+ | v2 |
Answer a question about mcpvessel's commands, flags, security model, or usage by searching its documentation on GitHub. Use this first.
Parameter descriptions lack actionable constraints. 'place' param in weather tools says 'A city or place name, for example "Lagos" or "Springfield, US"', includes example values that LLMs may reuse literally instead of adapting to context.
No error handling guidance. Tools provide no recovery hints (e.g., 'If place not found, try find_place() first'). Agents cannot self-correct on failures.