Loading ToolBench
Semantic code compiler for AI agents - transforms codebases into navigable concept graphs. MCP server that exposes code graph operations through the Model Context Protocol.
The server defines 13 tools with consistent naming (verb_noun pattern), reasonable descriptions (120-200 chars each), and complete JSON Schema input definitions for all tools. However, output schemas are not documented, error handling lacks recovery guidance, and several tools accept ID-only parameters where human-readable alternatives would improve usability. Parameter descriptions are present but terse (30-50 chars), leaving some ambiguity about constraints and expected formats. The tools follow a single responsibility principle well and are well-composed for semantic code analysis. No tool descriptions are under 20 chars, and all have type-annotated parameters, placing this solidly in the 'Fair to Good' range. The domain expertise is strong, but agent-facing clarity could be improved.
Analyze a symbol: retrieve its definition, references, and relationships
Get all code units defined in a specific file
Filter code units by programming language
Filter code units by type (function, module, class, etc.)
Find code units semantically similar to a given unit using embedding vectors
Get overview statistics about a code graph
Get the log of operations performed in this session
Get detailed information about a specific code unit
Output schemas not documented. Tool descriptions state what is returned (e.g., 'returns code units', 'returns statistics') but no formal schema defines the response structure, field names, or types. LLMs cannot reliably chain tools or extract specific fields from responses without explicit output documentation.
Parameter descriptions are terse (30-50 chars) and lack format/constraint guidance. For example, 'intent' parameter has description 'Extraction intent level' but does not explain what each enum value (exists, ids, summary, fields, full) returns or when to use each. 'min_similarity' lacks bounds (is 0 - 1 valid? what is typical?). These gaps force LLMs to guess parameter semantics.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 56 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 0 | - | v1 |
Analyze the impact of changing a code unit (what would break if this changes)
List all currently loaded code graphs
Load a code graph from .acb file
Execute a semantic or structural query on a code graph
Unload a code graph from memory
No error handling or recovery guidance documented. Tools accept ID parameters (unit_id, graph_name) but offer no guidance on what to do if the ID is invalid, the graph is not loaded, or the unit does not exist. Error messages and recovery steps are not visible in tool definitions.
Parameters accept IDs (unit_id, graph_name as string) without supporting human-readable alternatives. An agent calling find_similar_units must already possess a unit_id rather than a symbol name or file path. This forces extra lookup calls and breaks the chat data model where users refer to entities by name, not opaque IDs.
Tools returning lists (query_graph, filter_by_type, filter_by_language, filter_by_file, find_similar_units) accept max_results or top_k but lack pagination (no limit/offset, no next_cursor, no total count). If a query matches 1000 units, the agent cannot paginate, it gets a capped result set with no way to fetch more.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) declared. While all tools are marked READ_ONLY or REVERSIBLE in the provided metadata, these annotations are not visible in MCP protocol responses. The server should declare tool safety properties via the MCP annotation mechanism so clients can enforce guardrails.