Loading ToolBench
A comprehensive suite of three MCP servers: SQL Explorer (SQL Server database browsing and querying), Document Analyzer (PDF/DOCX/TXT/CSV text extraction and analysis), and Azure Git Explorer (Azure DevOps repository browsing)
18 tools with consistent naming (verb_noun pattern) and complete input schemas. All tools have descriptions (avg 180 chars, within baseline 194). However, descriptions are generic and lack LLM-optimized guidance on WHEN to use each tool vs alternatives. Parameter descriptions exist but are minimal (avg 40 chars vs baseline 72). No output schemas documented, LLMs cannot predict response structure. No error handling guidance. All tools are READ_ONLY, reducing risk, but composition chains are not optimized (e.g., search_code returns repo/path but get_file_content requires project/repo/path, missing project context).
Extract the full text of a document and return metadata. Supports PDF, DOCX, TXT, and CSV files. Args: file_path: Absolute or relative path to the file. Returns: A dict with ``file_name``, ``format``, ``size_bytes``, ``character_count``, ``page_count`` (PDF only), and ``text``.
Compare two documents and describe their similarities and differences. Args: file_path_1: Path to the first document. file_path_2: Path to the second document. Returns: A structured comparison (Markdown).
Show column details for a table including types, nullability, and key constraints. Args: database: Database name. table: Table name. May include schema prefix (e.g. ``dbo.Users``). Returns: A list of column descriptors.
Extract named entities from a document using GPT. Identifies people, organisations, dates, monetary amounts, locations, and other notable entities. Args: file_path: Path to the document. Returns: A dict mapping entity category to a list of extracted values.
Read the content of a file from a repository. Args: project: Project name. repo: Repository name. path: File path within the repo (e.g. ``src/main.py``). branch: Branch name (default ``main``). Returns: A dict with ``path``, ``branch``, ``content``, and ``size``.
No output schemas documented. LLMs cannot predict response structure, forcing them to guess field names and types. This breaks tool chaining (e.g., search_code returns 'repo' but get_file_content expects 'repo', unclear if they match).
Descriptions lack LLM-optimized guidance on WHEN to use each tool vs alternatives. E.g., 'List all Git repositories' does not explain when to call list_repositories vs get_repo_info. Agents waste reasoning cycles deciding between similar tools.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | B | 71 | 2026-07-28+ | v2 |
Get detailed information about a repository. Args: project: Project name. repo: Repository name. Returns: A dict with repository metadata including default branch, size, remote URL, and project info.
Preview a small sample of rows from a table. Args: database: Database name. table: Table name (may include schema, e.g. ``dbo.Users``). limit: Maximum rows to return (default 5, max 100). Returns: A list of row dicts.
List all branches in a repository. Args: project: Project name. repo: Repository name. Returns: A list of branch dicts with ``name``, ``commit_id``, ``is_base_version``, and ``creator``.
List all available databases on the SQL Server instance. Returns a list of database names the connected user can access. System databases (master, model, msdb, tempdb) are included.
List all projects in the Azure DevOps organisation. Returns: A list of project dicts with ``id``, ``name``, ``description``, ``state``, and ``url``.
List pull requests for a repository. Args: project: Project name. repo: Repository name. status: PR status filter -- ``active``, ``completed``, ``abandoned``, or ``all`` (default ``active``). Returns: A list of PR dicts with ``id``, ``title``, ``status``, ``created_by``, ``source_branch``, ``target_branch``, ``created_date``, and ``url``.
List all Git repositories in a project. Args: project: Azure DevOps project name. Returns: A list of repo dicts with ``id``, ``name``, ``default_branch``, ``size``, and ``web_url``.
List all user tables in the specified database. Args: database: Name of the database to inspect. Returns: A list of dicts with keys ``schema``, ``table``, and ``row_count``.
Execute a read-only SQL query and return the results. Only SELECT statements (including CTEs starting with WITH) are allowed. Dangerous keywords such as INSERT, UPDATE, DELETE, DROP, ALTER, EXEC, etc. are rejected before the query reaches the database. Args: database: The database to run the query against. query: A valid SELECT statement. Returns: A list of row dicts. Results are capped at ``max_rows`` (default 500).
Search for code across all repositories in a project. Uses the Azure DevOps Search API. Requires the *Code Search* extension to be installed in the organisation. Args: project: Project name. query: Text to search for (supports Azure DevOps search syntax). Returns: A list of match dicts with ``repo``, ``path``, ``branch``, and ``matched_lines``.
Perform a semantic search within a document. Finds passages relevant to the query and returns them with context. Args: file_path: Path to the document. query: The search query in natural language. Returns: Relevant excerpts and an explanation of their relevance.
Generate a SQL SELECT query from a natural-language question. This tool inspects the database schema and writes a query that answers the question. The generated query is **not** executed -- use ``run_query`` to execute it after reviewing. Args: question: A plain-English description of what data you want. database: The database to target. Returns: A SQL SELECT statement as a string.
Summarise a document in bullet points or as a paragraph. Args: file_path: Path to the document. style: ``"bullets"`` (default) or ``"paragraph"``. Returns: The summary as a string.
Tool composition broken: search_code returns 'repo' and 'path' but get_file_content requires 'project', 'repo', 'path'. The 'project' context is lost after search_code, forcing an extra lookup call.
Parameter descriptions are minimal (avg 40 chars vs baseline 72). E.g., 'project' is described as 'Azure DevOps project name' but does not clarify if it accepts project ID, project name, or both. Ambiguity forces LLMs to guess.
No error handling guidance. Tools like run_query and suggest_query can fail (invalid SQL, missing database) but descriptions do not explain recovery steps. Agents have no guidance on what to do if a call fails.