Loading ToolBench
MCP server that transforms codebases, documents, and papers into searchable bundles for LLM agents.
This server exposes 14 tools with mixed quality. Most have descriptions and basic parameters, but schema documentation is incomplete or missing for critical tools. Tool naming follows verb_noun patterns (preflight_list_bundles, preflight_create_bundle, etc.), which is good. However, several high-complexity tools like preflight_memory (12 parameters across 3 layers) and arxiv_search (15 parameters) lack detailed constraint documentation, forcing LLMs to infer valid combinations. Error handling and output schemas are not documented in the provided source. The memory tool has overlapping action enums that could cause confusion. Most tools score 50-70; the arxiv_search tool is better-specified than the core preflight tools.
Search arXiv papers by preset, query, or ID. Quick start: `{"preset": "ai_mainstream", "daysBack": 2, "brief": true}`. Get details: `{"idList": ["2301.07041"]}`. Tip: Use brief=true for listing (saves tokens), idList for full paper details.
Run duplicates, deadcode, circular dependency, and complexity checks.
Create a bundle from GitHub, local paths, PDFs, markdown, or web docs.
Delete a bundle.
Extract knowledge card (project summary, use cases, key APIs) from bundle.
Read OVERVIEW.md, START_HERE.md, and AGENTS.md for a bundle.
List available bundles and find a bundleId.
Input schemas completely missing for 6 tools (preflight_list_bundles, preflight_create_bundle, preflight_check, preflight_lsp, preflight_rag). These tools appear in toolRouter.ts with descriptions but no visible input parameter definitions.
preflight_memory tool combines 8 distinct actions (add, update, search, reflect, stats, list, delete, gc) with 3-layer semantics (episodic, semantic, procedural) into a single tool. This violates single-responsibility principle. LLMs must reason about valid action-layer combinations without explicit guidance. No error documentation for invalid combinations like add+procedural or reflect on episodic.
Output schemas not documented for any tool in provided source. arxiv_search description mentions 'paper count and titles' in brief mode and 'full paper details' in ID list mode, but response structure (fields, format, pagination, chaining IDs) not specified. Other tools give no hints about return values, field names, or downstream tool compatibility.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 46 | 2025-06-18+ | v2 |
| 2026-03-09 | F | 41 | 1.25.1+ | v1 |
Find definition, references, symbols, and diagnostics.
3-layer long-term memory system for persistent user context (ChromaDB). Layers: episodic (context quick injection), semantic (knowledge base/experience), procedural (global rules/preferences). Actions: add, search, list, update, delete, reflect, stats, gc.
Index/query semantic search content.
Manage ChromaDB: list/stats/inspect/delete indexed content, diagnose quality. Actions: collections, stats, list, sample, delete, delete_all, drop_collection, inspect, search_raw, diagnose.
Read specific file content from a bundle.
Inspect bundle directory structure.
Primary full-text search tool for code and docs.
preflight_rag tool has extremely vague description: 'Index/query semantic search content.' No explanation of when to use it, what 'content' means, how it differs from preflight_search_and_read (full-text) vs preflight_memory (long-term context), or what parameters it accepts. LLM cannot determine if it reads or writes.
arxiv_search parameter 'idList' and filter combinations (preset, query, daysBack, fromDate, toDate) lack documented interaction rules. Can both preset and query be provided? What happens if idList is set alongside filters? Description says 'preset overrides query' but doesn't clarify behavior with date filters.
preflight_delete_bundle is marked as DESTRUCTIVE but description is minimal: 'Delete a bundle.' No confirmation step, no dry-run mode, no recovery guidance. Given irreversible nature and agents' tendency to retry, missing error handling and confirmation pattern is a risk.
preflight_check tool description says it 'requires: path' but no documentation of what path format, whether absolute or relative, whether it must exist locally or in a bundle. bundleId is the primary MCP abstraction; requiring 'path' is inconsistent. Unclear if this operates on indexed content or fresh local filesystem.
Error handling and recovery guidance absent from all tool descriptions. No hints like 'If bundle not found, call preflight_create_bundle' or 'Search may timeout for large bundles, narrow query scope.' LLMs cannot self-correct on failures.