Loading ToolBench
A reference implementation for search-first tool discovery in multi-tenant environments. Demonstrates semantic tool discovery using embeddings and vector search, with tenant-scoped catalogs and SLA-aware tool selection.
The server exposes a single tool, search_tools, which is a meta-tool for discovering other tools rather than a domain-specific capability. The tool definition is visible in examples/search-tools-sketch.py with a description and parameter schema. However, critical quality gaps severely limit production readiness: (1) The tool accepts a 'db' parameter of type 'object' with no validation or schema, this is a code smell for passing raw database connections as parameters, violating secret-injection and input-validation patterns. (2) Parameter descriptions are present but minimal (e.g., 'Database handle (psycopg connection, etc.)' does not explain constraints or expected format). (3) No output schema is documented, the code returns a list of ToolDefinition dataclass instances, but the LLM has no visibility into the structure of returned objects (fields like 'sla', 'failure_modes'). (4) No error handling guidance, the stub embed() function raises NotImplementedError, and the SQL query has no error recovery or actionable messages. (5) The tool name 'search_tools' is generic and does not follow verb_noun conventions clearly (search_available_tools or discover_tools would be clearer). This is a reference implementation, not production code, and the quality reflects that.
The single entry point an agent calls to discover tools. Returns a list of ToolDefinition objects ranked by relevance based on semantic similarity to the query. Only tools enabled for the tenant are eligible.
Database connection passed as tool parameter (db: object). Violates secret-injection and input-validation patterns. Raw database handles should never be exposed as parameters, use server-side injection.
No output schema documented. Tool returns list[ToolDefinition] with fields (name, description, parameters, sla, failure_modes), but LLM has no visibility into structure. Agents cannot plan downstream tool calls without knowing response shape.
No error handling or recovery guidance. embed() raises NotImplementedError; SQL query has no error handling. LLM receives no actionable error messages if embedding fails or query times out.
Parameter 'db' lacks type constraints and validation rules. Description says 'Database handle (psycopg connection, etc.)' but does not specify required methods, connection state, or error conditions.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 50 | 2026-07-28+ | v2 |
Tool name 'search_tools' is generic and does not clearly convey action. 'discover_available_tools' or 'search_tool_catalog' would be more explicit and help LLMs disambiguate intent.