Loading ToolBench
A multi-agent system for answering questions about Python codebases using MCP protocol. Consists of 5 specialized agents (Orchestrator, Indexer, Graph Query, Code Analyst, Monitoring) coordinated via FastMCP.
28 tools with mixed quality. Most have basic descriptions (50-150 chars) and simple input schemas, but lack depth. Parameter descriptions are minimal or absent. Output schemas are undocumented. No tool annotations (readOnlyHint/destructiveHint). Error handling guidance is absent. Tool names follow verb_noun convention (good), but many descriptions are too generic to guide LLM selection. Composition is reasonable (tools are single-purpose), but chaining support is weak, responses don't consistently include IDs needed by downstream tools.
Comprehensive class analysis including methods, inheritance, and patterns.
Deep analysis of a function's logic, complexity, and behavior.
Classify query intent and extract key entities.
End-to-end chat: analyze the query, route to agents, call them, and synthesize.
Compare two code entities.
Run a custom Cypher query with safety constraints (read-only, max 1000 results).
Generate explanation of how code works.
No tool annotations (readOnlyHint/destructiveHint/idempotentHint). LLMs cannot distinguish safe read-only tools from destructive ones (index_repository, set_user_preferences). This violates pattern:command-tool and increases risk of unintended side effects.
Output schemas are undocumented. Tools like analyze_function, get_dependencies, and execute_query return complex nested structures, but the response format is not specified. LLMs cannot plan downstream tool calls without knowing what fields to expect.
Parameter descriptions are minimal or missing. 'scope' in find_patterns is described as 'Scope to search for patterns (e.g., class name, module name, or global)', no guidance on format, constraints, or what happens if scope is invalid. Similar gaps in entity_name, relationship_type, cypher parameters.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 53 | <=2025-11-25 | v2 |
Extract code entities and their relationships from a file.
Locate a class, function, method, or module by name.
Detect design patterns in code.
Get entities related to the given entity by a specific relationship type.
List active conversation sessions with message counts.
Get per-agent latency and success/failure counts from recent conversations.
Extract code with surrounding context.
Retrieve conversation history and context for a session.
Find what an entity depends on (imports, base classes, called functions).
Find what depends on an entity (who imports, inherits, or calls it).
Report knowledge graph size and node/relationship counts.
Report indexing progress for a background job.
Retrieve query performance statistics from recent sessions.
Check health of all infrastructure services (Neo4j, Redis).
Re-index a single Python file. Clears old data for this file first.
Full repository indexing. Clones repo, parses all Python files, populates Neo4j.
Parse a Python file and return a simplified AST representation.
Determine which agents should handle the query based on analysis.
Store user preferences for a session (e.g. verbosity, focus area).
Combine agent outputs into a coherent response.
Follow the import chain from a module up to a given depth.
No error handling guidance. Tools like execute_query (Cypher) and index_repository (git clone) can fail in many ways (syntax error, network timeout, permission denied), but no recovery hints are provided. LLMs will not know whether to retry, ask the user, or abort.
Weak tool chaining support. Response schemas do not document which IDs/references are returned. E.g., find_entity should return file_path and line_start so get_code_snippet can be called next, but this is not explicit. Forces LLMs to infer or make extra discovery calls.