Loading ToolBench
Provides an abstract interface that allows LLMs to connect to fact sources such as LSPs, code diagnostics, symbol definitions/references, links, and frontmatter
OpticLM Connector provides 23 tools across code navigation, file manipulation, and workspace analysis. Naming is generally strong and verb-driven (goto_definition, find_references, apply_edit). Descriptions are present and adequate for most tools, ranging from 40-350 chars. Schemas are consistently defined with proper JSON Schema structure including types and descriptions. However, several tools have incomplete or vague descriptions, parameter relationships are underdocumented, and error handling guidance is minimal. Output schemas lack documentation in descriptions, callers must infer structure. The apply_edit tool has a detailed but procedural description; others like get_outline and get_diagnostics are terse. Tool composition is sound (each does one thing), but discovery is hindered by lack of chaining hints (e.g., 'Call request_file first to get the hashline format needed for apply_edit').
Add a link to a document by finding a pattern and replacing it with a link.
Apply a text edit to a file. WORKFLOW: First read the file via the request_file tool to get hashline-formatted content (e.g., "3:a1| return x"). Then reference lines to specify the edit range. The line:hash prefix verifies the file has not changed since your read - if it has, the edit is rejected and you must re-read the file. For single-line edits, only start_line is needed. For multi-line edits, provide both start_line and end_line. The edit replaces the entire line range (inclusive) with replace_text.
Get call hierarchy for a function or method. Shows incoming or outgoing calls.
Resource that provides diagnostics formatted as markdown for all files in the workspace.
Resource that provides filesystem access with support for reading files and directories.
Find all references to a file across the workspace. Returns locations that import or link to the given file.
Sparse descriptions for read-only discovery tools (get_outline, get_diagnostics, get_workspace_diagnostics, get_backlinks, get_outlinks, get_link_structure). These are 25-60 chars, below the 50-char minimum for LLM decision-making. Missing context on when to call them vs alternatives and what structure they return.
Output schemas are not documented in tool descriptions. Callers must infer structure (e.g., apply_edit returns { updated: ... }, goto_definition returns { snippets: ... }). LLMs cannot plan downstream calls without knowing the response shape.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 64 | 2025-06-18+ | v2 |
Find all references to a symbol. Returns a list of locations where the symbol is used.
Resource that provides frontmatter data for documents.
Get incoming links (backlinks) to a specific file.
Get diagnostics (errors, warnings, hints) for a specific file.
Get all frontmatter for a specific document.
Get the frontmatter structure for a specific property across documents.
Get all links in the workspace.
Get the document outline (symbols like classes, functions, variables) for a specific file.
Get outgoing links from a specific file.
Get all diagnostics (errors, warnings, hints) across the entire workspace.
Perform a global find operation across the workspace.
Navigate to the definition of a symbol.
Navigate to the type definition of a symbol.
Resource that provides workspace link structure data.
Resource that provides document outline (symbols) formatted as markdown for a specific file.
Access filesystem resources. For directories: returns children as an array (git-ignored files excluded). For files: returns content in hashline format where each line is prefixed with "<lineNumber>:<hash>|" (e.g., "1:a3|function hello() {"). The hash is a 2-char hex CRC16 digest of the line content. Use these line:hash references with the apply_edit tool to make edits. Supports optional line range filtering (start_line/end_line) and regex pattern filtering.
Set a frontmatter property on a document.
Undocumented parameter relationships. line_hint and order_hint (FuzzyPosition) are optional disambiguation fields used across 5 tools (goto_definition, goto_type_definition, find_references, call_hierarchy, find_file_references), but the descriptions do not explain when both are needed, how they interact, or what happens if neither is provided.
apply_edit has a complex multi-step workflow (read file → check hashline format → provide line:hash prefix) documented in the description. However, no recovery guidance is provided if hashline validation fails or if the file changes mid-edit. Error handling only says 'the edit is rejected', does not tell LLM how to recover.
No pagination documented for tools that return lists (find_references, call_hierarchy, find_file_references, global_find, get_workspace_diagnostics). Input schemas show no limit, offset, or cursor parameters. Responses could be unbounded, risking context window overflow.
Tools accepting file paths (request_file, get_diagnostics, get_outline, get_outlinks, get_backlinks, get_frontmatter, set_frontmatter, add_link) document paths as 'relative file path' but do not specify allowed characters, length limits, or path traversal guards. Descriptions should include constraints like 'Must not contain .. or absolute paths.'
Error handling is generic ('error: message'). No guidance on retry strategy, categorization (retryable vs user-fixable vs fatal), or actionable recovery steps. E.g., if file not found, should LLM call request_file to discover paths? If symbol not found, should it search globally?
Tool composition hints are missing. E.g., apply_edit's description explains the workflow (read file → check hashline → edit) but does not explicitly state 'First call request_file to get hashline content.' Similarly, tools that accept symbol_name do not hint at calling global_find or get_outline to discover available symbols.
Resources (diagnostics, outline, frontmatter, graph, filesystem) lack parameter validation guidance. E.g., 'graph' resource accepts no parameters and presumably returns all workspace links, no pagination control, no filtering, no size cap documented. Large workspaces could return millions of links.
set_frontmatter accepts a 'value' parameter of type ['string', 'number', 'boolean', 'array', 'object', 'null']. No validation constraints documented. Can an LLM pass arbitrary nested objects? Are there size limits? Structure constraints?