Loading ToolBench
Repository knowledge engine with grounded Q&A and MCP server. Exposes ask_project and refresh_project as MCP tools for querying project knowledge base.
RepoBrain exposes 26 tools across multiple files with HIGHLY INCONSISTENT quality. Tools present visible input schemas with parameter descriptions, but many lack critical detail. Descriptions range from adequate (ask_project, search_code) to minimal (greet_user, reverse_text). Several tools are demo/example implementations mixed into production registration. Output schemas are entirely undocumented. Error handling is absent or minimal. Schema validation shows parameters have types, but constraints are rarely enforced (no enums for categorical inputs like file_type, mode). The server's strength (code-exploration tools) is undermined by weak composition (26 separate tools when many could consolidate), and critical tools like run_python_code expose dangerous capabilities with minimal safeguards. This is a C-grade server: good intent, significant structural gaps.
Ask a question using graph-first, all-file project context.
Answer a question about the project using the knowledge hub. Searches the codebase, reads actual source files, checks git history, and synthesizes a grounded answer with file paths and line numbers. Use this instead of manual grep or file reading.
Safely evaluate a mathematical expression and return the numeric result. Parses the expression using Python's AST and permits only a restricted set of nodes and operators.
Performs a deep research analysis on the given topic.
Get detailed help for a specific MCP tool.
Retrieves the current stock price for a given ticker.
Output schemas entirely undocumented. No tools declare structured return types, LLMs cannot plan downstream tool calls or extract specific fields. All 26 tools affected.
Demo and example tools mixed into production tool set. Tools like 'greet_user', 'reverse_text', 'web_search' (mock), 'get_stock_price' (mock), 'get_weather' (placeholder) and 'send_email' (simulated) are not functional implementations, they pollute the tool namespace and confuse agents. Agents cannot distinguish real vs stub tools.
search_by_type accepts 'file_type' parameter with description mentioning 'code/documentation/data/media/binary' but NO enum constraint. LLMs will hallucinate invalid values like 'source', 'docs', 'image' instead of the exact keywords.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 66 | <=2025-11-25 | v2 |
Return mock weather data for a given city. Placeholder that demonstrates a typical tool shape.
Greets the user by name with a friendly message. Demonstration tool showing how new tools can be added without modifying agent.py.
Initialize a new project from the template in quick or full mode.
List the contents of a project directory.
List all configured MCP servers and their connection status. Provides information about which MCP servers are configured, whether they are connected, and how many tools each server exposes.
List all available tools from MCP servers.
Perform a health check on all MCP connections. Tests the connectivity to all configured MCP servers and returns a summary of their status.
Retrieve a relevant semantic subgraph for a user query. Returns a tool-friendly structure with summary, triples, evidence, nodes and edges.
Read a file from the project, returning numbered lines.
Read lightweight metadata for a file without loading full content.
Read markdown memory content for model inspection.
Refresh graph-first knowledge artifacts for all file types.
Rebuild the project knowledge base (.repobrain/conventions.md and structure.md). Run this after significant code changes to keep the knowledge base up to date.
Reverses the given text string. Demo tool showing multiple tools can be added in one file.
Execute Python code using the configured sandbox.
Find files by high-level type: code/documentation/data/media/binary.
Search project source files for a text pattern.
Search markdown memory using ripgrep with a Python fallback.
Mock sending an email to demonstrate a stateful side-effect tool. Does not actually send email; simulates the action.
Performs a web search for the given query. Mock function to demonstrate the expected structure of a tool.
run_python_code tool marked IRREVERSIBLE with minimal safeguards. No dry-run option, no confirmation step, no explicit permission gates. Description does not warn of catastrophic risk. Agents can trivially execute arbitrary code including file deletion, network calls, resource exhaustion.
Error handling absent or implicit. Tools return string error messages ('Error: path outside project', 'No results found') rather than structured error categories (retryable, user-fixable, fatal) that guide agent recovery. LLMs receive unstructured text and must guess the next action.
Pagination absent on list/search tools. search_code hardcoded to _MAX_SEARCH_RESULTS=50 without offset/limit parameters. Large projects will truncate results silently; agents have no way to fetch subsequent pages. Violates pattern:paginated-result.
Tool naming clarity issues: 'query_graph' vs 'ask_filesystem' vs 'ask_project', LLMs cannot distinguish when to use each. Missing usage guidance. 'refresh_project' vs 'refresh_filesystem', are these idempotent or not? Descriptions don't state.
Tools like deep_research, init_agent_repo, and MCP introspection tools (list_mcp_servers, list_mcp_tools) lack documented return structures. Deep_research promises 'deep research analysis' but no schema of what 'analysis' contains.
Path traversal safeguards present (is_safe_path checks in read_file/list_directory) but not uniformly applied. search_code walks the tree with should_skip_dir filtering, but if a tool receives an untrusted workspace path, the validation is incomplete.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite CURRENT MCP spec supporting them. Agents cannot determine which tools are safe to retry, which require confirmation, or which are read-only without parsing descriptions.