Loading ToolBench
MCP server for .NET code analysis using Roslyn, providing tools for analyzing solutions, projects, files, and discovering code symbols and references.
This server demonstrates basic tool structure with proper naming conventions and functional descriptions, but falls significantly short of production-grade quality. All 7 tools follow the verb_noun pattern (Load*, Analyze*, Get*, Find*), which is a positive. However, critical gaps emerge in parameter descriptions, schema documentation, and error handling. The server provides inline descriptions for individual parameters (e.g., 'Path to the .NET solution file'), but lacks crucial guidance on formats, constraints, dependencies, and recovery paths. Output schemas are not documented, LLMs cannot see what fields to expect from responses, forcing them to guess at result structure. Error handling is present in the code (try/catch wrapping with JSON error responses), but the error messages are generic ('Failed to load solution') rather than actionable recovery guidance. All tools are READ_ONLY, which is appropriate, but no tool explicitly declares this risk posture via annotations (toolAnnotations=false). The symbol-based tools (GetSymbols, FindReferences) accept optional parameters but do not document mutually exclusive patterns or format constraints (e.g., what constitutes a valid 'symbolKind' enum, what 'symbol' format is expected, simple name vs. fully qualified name?). Parameter descriptions are present but average ~20-30 characters, which is below the 72-char baseline for A+ tools and lacks context on format/range.
Analyzes a specific file and returns detailed information about symbols, diagnostics, and structure.
Analyzes a project and returns comprehensive analysis results including diagnostics, symbols, and metrics.
Analyzes a solution and returns analysis for all projects within it.
Finds all references to a symbol across the loaded workspace. Optional projectName to limit scope.
Gets all symbols of a specified kind from a project. Optional symbolKind parameter (Class, Method, Property, Field, etc.).
Loads a single .NET project and prepares it for analysis.
Loads a .NET solution and prepares it for analysis. Returns information about loaded projects.
Output schemas are completely undocumented. Code shows ExecuteAsync returns JSON via JsonConvert.SerializeObject, but LLMs have no declared schema for response structure. For AnalyzeProject, the response includes fields like 'projectName', 'totalLines', 'sourceFileCount', 'diagnostics' (as array of objects with 'id', 'severity', 'message', 'location'), 'sourceFiles', 'dependencies', 'analyzedAt', but this structure is buried in implementation code, not in a tool schema contract. LLMs cannot reliably parse or chain results without documented schemas.
Parameter descriptions lack constraints, formats, and dependencies. Example: GetSymbols accepts optional 'symbolKind' parameter described as 'Kind of symbol to filter by (Class, Method, Property, Field, etc.)' but does not declare valid enum values, does not explain if the parameter is case-sensitive, does not document what happens if an invalid kind is passed. FindReferences accepts 'symbol' parameter with description 'Name or fully qualified name of the symbol to find references for', but does not clarify: is 'MyClass' valid or must it be 'MyNamespace.MyClass'? What about nested classes? Partial matches? This ambiguity will cause LLMs to guess and fail.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 40 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 8 | - | v1 |
Error responses are generic and non-actionable. Code returns JSON like { 'success': false, 'error': 'Failed to load solution' } or { 'success': false, 'error': 'Project not found after loading' }. These tell the LLM nothing about recovery. Per the rubric, errors should guide the agent: 'Solution file not found at /path/to/file.sln. Check the path and retry, or call ListSolutions() to discover available solutions.' Current errors do not suggest next steps, leaving agents stuck.
Optional parameters lack dependency/relationship documentation. GetSymbols has optional 'symbolKind' and optional 'projectName'. Are both optional independently, or does one depend on the other? FindReferences has optional 'projectName'. Does omitting 'projectName' search across all loaded projects, or only the most recently loaded one? Undocumented dependencies cause silent misuse.
No tool declares permissions or access scope. Per the rubric, each tool should declare required permissions (e.g., 'read:project', 'write:file'). This prevents least-privilege configuration and leaves audit trails vague.
No tool includes tool annotations (readOnlyHint, destructiveHint, idempotentHint). Server declares toolAnnotations=false. All tools are READ_ONLY and idempotent, but LLMs have no machine-readable way to know this. Adding readOnlyHint=true would enable agents to safely retry without fearing side effects.
Path parameters accept raw file paths with no validation, normalization, or sanitization guidance. A malicious or confused LLM could pass a path like '../../../sensitive_file.sln' or a URL. Code does not document: must paths be absolute or relative? Are symlinks followed? What about UNC paths on Windows? These details matter for both security (path traversal) and correctness (resolution).