Loading ToolBench
MCP server for managing collections, embeddings, summaries, and extensions. Provides tools for semantic search, long-term memory storage, table of contents generation, and integration with external extensions.
Server has 10 tools with mixed definition quality. All tools have descriptions and input schemas are present, but many are incomplete or lack proper parameter-level descriptions. Naming is generally clear and action-verb-based (health_check, add_fact, query_collection), but several tools have vague or incomplete parameter documentation. Output schemas are not documented in the tool descriptions, making it difficult for LLMs to reason about return types. Error handling is minimal, most tools return simple dicts without actionable recovery guidance. Parameter descriptions often reference function arguments but don't explain constraints, formats, or valid ranges. The call_extension tool accepts 'input' as a string (not properly typed) which is a significant schema violation.
Saves a fact about the user for long-term memory. - fact: The full text of the fact to remember. - summary: A brief summary of the fact, used for search embeddings.
Add TOC for the collection - collection_name: name of the ChromaDB collection - toc: table of contents content
Adds a fact to a given collection. - collection_name: The name of collection to save fact - fact: The full text of the fact to remember. - summary: A brief summary of the fact, used for search embeddings.
Calls a command on a connected extension. - id: The ID of the extension to call. - name: The name of the command to invoke. - input: A dictionary with the input parameters for the command.
Returns a list of enabled collections with their names, IDs, and descriptions.
Returns a list of all connected extensions and their supported commands.
call_extension's 'input' parameter is typed as string instead of object, forcing LLMs to manually JSON-serialize dict arguments. This violates proper parameter typing and will cause serialization errors.
No tool returns documented output schemas. LLMs cannot infer what fields to expect or plan downstream tool calls. For example, collection_list returns a list of dicts with 'id', 'name', 'description', 'properties', but this structure is not documented in the tool schema.
Parameter 'input' in call_extension lacks any specification of its expected structure. Description says 'A dictionary with the input parameters for the command' but provides no schema, format hints, or validation guidance. LLMs will struggle to construct valid inputs.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 61 | <=2025-11-25 | v2 |
| 2026-03-09 | C | 60 | - | v1 |
Retrieves one or multiple chunks from a ChromaDB collection using IDs. - collection_name: name of the ChromaDB collection - ids: a single ID (string), a list of IDs (list[str]), or a JSON string representation of a list
Retrieves TOC for the collection - collection_name: name of the ChromaDB collection
Returns the server name and a health status.
Queries a collection with a given text. - collection_name: The name of the collection to query. - query_text: The text to query the collection with. - n_results: The number of results to return.
get_chunks_by_id parameter 'ids' lacks a type definition in the schema (marked with no type, only description). This is a schema violation, LLMs cannot determine whether to pass a string, list, or JSON string.
Error handling is minimal. Tools return {'status': 'error', 'message': '...'} without categorizing errors as retryable vs. fatal or providing recovery guidance. LLMs cannot determine if a failure is transient or requires user intervention.
No parameter constraints documented. query_collection's 'n_results' has no minimum/maximum bounds, LLMs may pass absurd values (0, 1000000). add_fact's 'fact' and 'summary' have no length limits documented.
collection_name appears as a required parameter in multiple tools but no enum or discovery tool hints when a user passes an invalid name. Error handling does not provide 'Did you mean?' suggestions for typos.
Descriptions are inconsistent in length and detail. health_check (40 chars) is minimal but adequate; add_fact (114 chars) includes parameter details inline, which works but is not ideal for schema clarity. Average description length is ~65 chars, below the 194-char baseline for production tools.
Tools that modify state (add_fact, add_to_collection, add_table_of_contents, call_extension) do not explicitly state in descriptions that they are destructive/write operations. This makes it harder for LLMs to reason about side effects and idempotency.