Loading ToolBench
An MCP server for semantic code analysis, structural pattern matching, and AI-assisted code reasoning with support for local inference and workspace indexing
context-sherpa is a Go MCP server for semantic code analysis. It exposes 9 tools with basic descriptions but significant quality gaps. Tool naming is verb-based and generally clear (query_local_reasoning, list_local_models, switch_local_model, etc.), meeting baseline naming conventions. However, parameter descriptions are sparse or missing, schemas lack type definitions visible in source, and output schemas are not documented. The server provides tool descriptions of moderate length (19-100+ chars), but most parameter descriptions are minimal. No evidence of error handling guidance, validation rules, or security considerations like API key injection. The tools appear genuinely registered in pkg/mcp/server.go, so they are not inferred, but the limited source code visibility prevents full assessment of schema completeness. Averaging across 9 tools with descriptions present but incomplete schemas and weak parameter annotation yields a below-average definition quality score.
Scans a list of SCIP references to identify which call sites are most likely to be affected by a change.
Validates a code snippet against the project-specific rules/ directory.
(The Router) Determines which Sherpa tool (Symbolic, Structural, or Semantic) is best suited for a user's high-level query.
Translates natural language into a valid ast-grep S-expression or pattern.
List available models from the configured local inference engine (Ollama/LM Studio).
Requests the local inference engine (Ollama/LM Studio) to download a new model.
No visible input schema definitions in source code. Schemas are inferred from parameter objects in the provided source, but complete JSON Schema with type, minimum, maximum, and enum constraints is not shown. Cannot verify full schema compliance.
Parameter descriptions are minimal. Examples: 'Optional model ID to use. Defaults to currently active.' (modelId in query_local_reasoning) and 'The question or instruction for the local SLM to reason about.' (prompt). Most lack detail on format, range, and allowed values. LLMs cannot infer valid input ranges or constraints.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-21 | D | 54 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 27 | - | v1 |
(The Fallback) A catch-all tool for asking any open-ended semantic question about a code snippet that doesn't fit a specific template.
Distills raw code into a 3-sentence functional summary (Inputs, Outputs, Side-effects).
Changes the preferred model in the Hub settings.
No documented output schemas. Tools like query_local_reasoning, summarize_code_intent, and classify_repo_intent do not show what fields or structure they return. LLMs cannot plan downstream tool calls or extract the right fields without output documentation.
No error handling or recovery guidance visible. Tools do not document what happens on failure (model not found, SCIP parse error, rule file missing) or how LLMs should respond. Error responses cannot guide the agent to next steps.
No tool annotations present. Tools like switch_local_model and pull_inference_model modify state (WRITE risk) but are not marked with destructiveHint or idempotentHint. Tool annotations are now part of the current MCP spec (2026-07-28) and are missing.
Tool descriptions lack guidance on when to call each tool versus alternatives. For example, query_local_reasoning is labeled '(The Fallback)' and classify_repo_intent is labeled '(The Router)', but the description does not explain how an LLM should choose between them. Insufficient discrimination for multi-tool selection.