Loading ToolBench
The server defines 3 tools with reasonable descriptions and basic schemas. Tool names follow verb_noun conventions (search_code, index_codebase, index_status). Descriptions are moderately detailed (100-250 chars) and explain use cases, but lack explicit guidance on when to use each tool vs alternatives. Input schemas are present and typed, but parameter descriptions are minimal or missing context about constraints and formats. Output schemas are partially documented via return type hints (SearchResponse, IndexCodebaseResponse, etc.) but the actual field structures are not visible in the provided source. Error handling returns structured ErrorResponse objects, which is good, but recovery guidance is minimal. The server lacks output schema documentation that LLMs need to plan downstream operations.
Index a codebase for semantic search. Scans Python files, extracts functions/classes/methods, generates embeddings, and stores them for fast semantic search. Use force=True to re-index everything even if files haven't changed. Otherwise, only new and modified files are indexed (incremental).
Get the index status for a project. Returns information about whether the project is indexed, when it was last updated, and how many files and chunks are indexed. Note: search_code automatically re-indexes stale files before searching, so there is no need to check or act on staleness manually.
Search for code semantically similar to the query. Finds code by meaning, not just text matching. Use this when you want to find code related to a concept without knowing exact variable/function names. Examples: - "authentication logic" - finds login, session handling, token validation - "error handling for API calls" - finds try/except blocks, error responses - "database connection setup" - finds connection pooling, ORM initialization Automatically indexes the project if not already indexed, and re-indexes any files that have changed since the last search.
Output schemas not documented in tool descriptions. LLMs cannot infer what fields SearchResponse, IndexCodebaseResponse, and IndexStatusResponse contain, limiting ability to chain tools or extract specific data.
Parameter descriptions lack format constraints and validation rules. The 'query' parameter has minimal description; 'project_path' does not specify that it must be absolute or detail error handling for invalid paths. 'limit' defaults to 10 but lacks min/max bounds or guidance on acceptable range.
Error handling provides ErrorResponse but recovery guidance is sparse. When project_path does not exist, the tool returns a plain error message without suggesting alternatives or next steps (e.g., 'Call search_users to find valid projects').
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 8 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 42 | - | v1 |
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) present. index_codebase modifies state (writes to index) but this is not declared in the schema, forcing LLMs to infer safety properties from the description alone.
Result limits not enforced or documented. search_code accepts a 'limit' parameter but no maximum bound is stated. If an LLM passes limit=10000, the response could exceed context windows. The description should state: 'Maximum results capped at 100 to prevent context overflow.'