Loading ToolBench
MCP server for codebase intelligence - tree-sitter indexing, PageRank, blast radius
Qartez MCP has 8 tools with schemas present in test files, but exhibits significant definition quality issues. Tool names lack consistent verb prefixes (qartez_tools, qartez_boundaries, qartez_hierarchy are not action-verb-first). Descriptions vary wildly in quality, some are adequate (20-80 chars) but most lack clarity on WHEN to use the tool or what differentiates it from similar tools. Parameter descriptions are present but often generic ('Symbol name to analyze' lacks specificity about valid formats, length, or examples). No output schemas are documented, making it impossible for LLMs to plan downstream calls. Error handling is not evident in the visible code. Tools like qartez_move, qartez_rename_file, and qartez_replace_symbol are destructive/irreversible but lack dry-run or confirmation patterns. The 'apply' boolean parameter appears across refactoring tools but is not consistently explained as a safety gate. Schema definitions are visible in test files (not primary source), suggesting tool registration may be inferred rather than explicit, per the hard rules, this caps per-tool scores at 50.
Tool for managing code boundaries with auto-clustering support and file writing
Analyze type hierarchies with configurable depth and output formats
Refactor tool to move symbols between files with validation
Refactor tool to rename files with validation
Refactor tool to replace symbol definitions with validation
Analyze symbol deletion safety with per-symbol reference counting
Discover and enable additional Qartez tools. Call with no arguments to see all available tiers and tools. Use enable/disable to dynamically add or remove tool tiers or individual tools. Tier names: 'core' (always on), 'analysis', 'refactor', 'meta'. Pass 'all' to enable everything.
Tool names lack consistent action-verb prefixes. Only 'qartez_move', 'qartez_rename_file', 'qartez_replace_symbol', and 'qartez_safe_delete' hint at actions; others like 'qartez_tools', 'qartez_boundaries', 'qartez_hierarchy' are nouns. LLMs rely on verb-first naming to infer tool purpose before reading descriptions.
No output schemas documented for any tool. LLMs cannot plan downstream tool chains or extract required fields (e.g., if qartez_hierarchy returns symbol IDs needed by qartez_move). This breaks composition (pattern:tool-chain) and forces agents to guess return structure.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 46 | 2026-07-28+ | v2 |
Generate wiki documentation with clustering and configurable output
Three destructive/irreversible tools (qartez_move, qartez_rename_file, qartez_replace_symbol) expose 'apply' boolean parameter, but it is not documented as a dry-run or confirmation gate. Description text does not explain that 'apply=false' is a safety preview. Pattern confirmation-request requires explicit guidance on safe/unsafe modes.
Parameter descriptions lack specificity. 'Symbol name to analyze' (qartez_hierarchy) does not state: What constitutes a valid symbol? File extension? Scope qualification (namespace::symbol)? Regex pattern? LLMs cannot infer constraints from vague descriptions and will pass malformed input.
Tool definitions are sourced from test files (src/tests/fp_regression_*.rs), not primary implementation. Hard scoring rule: if tool registration is inferred rather than explicitly visible in main code, cap per-tool scores at 50. Actual tool registration code should be in src/main.rs or src/server/ with explicit schema and description declarations.
No error handling guidance documented. Destructive tools provide no recovery paths (e.g., if qartez_move fails mid-operation, can it be rolled back? Does qartez_rename_file offer undo?). Per pattern:recovery-guide, error responses must tell the LLM what to do next.
Tool descriptions inconsistently explain WHEN to use a tool vs. similar alternatives. E.g., qartez_hierarchy and qartez_boundaries both analyze codebase structure, but when does an agent call one vs. the other? Missing dependency hints and sequencing advice.
Several parameters accept file paths or symbolic identifiers without documented validation. E.g., 'write_to' path in qartez_boundaries and qartez_wiki, is it an absolute path? project-relative? Does it auto-create directories? Can it traverse parent dirs (security risk)? Path traversal must be explicitly guarded.