Loading ToolBench
Self-hosted connector gateway: catalog, config management, OAuth connections and aggregated MCP
connect-it exposes 12 tools with basic HTTP transport and MCP SDK integration. All tools have descriptions (10-50 chars, mostly short) and input schemas with type definitions. However, descriptions are minimal and lack LLM-optimization guidance (e.g., 'Health check' is 12 chars; rubric baseline is 194 chars avg). Parameter descriptions exist but are terse ('connector_type', 'New password, at least 8 characters'). No output schemas documented. No error handling guidance, recovery hints, or actionable error messages visible. No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear risk levels (READ_ONLY, WRITE, DESTRUCTIVE). Security concerns: login/changePassword expose credentials as parameters (anti-pattern). No pagination support visible for list tools. Composition is reasonable (separate tools per action), but descriptions do not guide LLM selection or explain when to use each tool.
List every connector for the admin UI, cookie authenticated
Change the administrator password
Delete connector config
Read connector config; secrets echo back only the keys that are set
Config form metadata for a connector, derived from the ConfigFields of its Definition
Get a single connector
Health check
Auth methods and credential fields of a connector, for the create-connection form
Credentials exposed as tool parameters: login() accepts username/password, changePassword() accepts password. Credentials must never appear as parameters, use server-side secret injection via environment variables or vault. Agent traces log every parameter, leaking secrets into logs and prompt history.
Descriptions are too short and lack LLM-optimization guidance. 'Health check' (12 chars), 'Log in as administrator and receive a session cookie' (50 chars), 'List every connector with its runtime status' (43 chars). Rubric baseline is 194 chars avg. Descriptions should explain WHAT, WHEN to use, and any prerequisites. Short descriptions force LLMs to guess when to select each tool.
No output schemas documented. LLMs cannot plan downstream tool calls or extract required fields without knowing what each tool returns. E.g., does listConnectors return connector_id, name, status? Does getConfig return public and secrets separately?
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 51 | 2026-07-28+ | v2 |
List every connector with its runtime status
Log in as administrator and receive a session cookie
Write connector config; public is replaced wholesale, secrets are merged and an empty string deletes
Enable or disable a connector
No tool annotations despite clear risk levels. Tools marked DESTRUCTIVE (deleteConfig), WRITE (changePassword, updateConnectorEnabled, putConfig), and READ_ONLY lack readOnlyHint, destructiveHint, or idempotentHint annotations. These guide LLM safety reasoning and retry logic.
No error handling guidance or recovery hints. If a tool fails, LLMs have no actionable next step. E.g., if getConnector fails with 'connector not found', should the LLM call listConnectors first? If putConfig fails with 'if_match mismatch', should it retry with getConfig to fetch the current timestamp?