Loading ToolBench
MCP server for querying SAS XPT data sources via SQL
The SAS XPT MCP server provides three read-only data exploration and query tools with basic schemas and descriptions. Tool naming follows verb_noun conventions (get_tables, get_columns, run_query), which is positive. However, descriptions are moderately detailed but lack actionable guidance on error conditions and recovery. Parameter schemas are present and typed, but lack detailed descriptions and validation constraints. Output format (CSV) is mentioned but no structured schema is documented. Error handling is minimal, tools throw RuntimeExceptions with basic messages rather than providing actionable recovery guidance. The server demonstrates competent basic structure but lacks the refinement expected for production-grade agent tools. Code shows proper use of the MCP SDK (0.8.1), but implementation falls short of pattern best practices in documentation, validation, and error guidance.
Retrieves a list of fields, dimensions, or measures (as columns) for an object, entity or collection (table). Use the `{prefix}_get_tables` tool to get a list of available tables. The output of the tool will be returned in CSV format, with the first line containing column headers.
Retrieves a list of objects, entities, collections, etc. (as tables) available in the data source. Use the `{prefix}_get_columns` tool to list available columns on a table. Both `catalog` and `schema` are optional parameters. The output of the tool will be returned in CSV format, with the first line containing column headers.
Execute a SQL SELECT statement.
No output schema documentation. Tools return CSV-formatted text, but LLMs receive no schema definition for the structured results. Downstream tool selection and data extraction become error-prone.
Minimal error handling and no recovery guidance. Tools throw RuntimeException('ERROR: ' + ex.getMessage()) without categorizing errors as retryable, user-fixable, or fatal. LLMs receive no guidance on what to do next.
Parameter descriptions are sparse. 'catalog' and 'schema' parameters are simply described as 'The catalog name' and 'The schema name' without explaining optionality, format, allowed values, or when to use them. The run_query 'sql' parameter mentions SQL dialect details inline but lacks structure.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 49 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 46 | 0.8.1+ | v1 |
No input validation or constraint enforcement. Parameters lack regex patterns, enums, min/max bounds, or length limits. The sql parameter in run_query accepts any string; no validation against allowed clauses (FROM, JOIN, GROUP BY, etc.) occurs before execution.
Vague descriptions for run_query. The tool description 'Execute a SQL SELECT statement' is minimal (34 chars, below the baseline average of 194 chars). It mentions valid clauses but doesn't explain what the tool is FOR or WHEN to use it vs. other data access patterns.
No pagination support. If a table has thousands of rows, run_query returns all results (CSV), which could exhaust token budgets and degrade LLM reasoning. No LIMIT/OFFSET guidance or result count in response.
Tool composition issue: get_tables and get_columns descriptions reference each other ('Use the get_tables tool...') but don't explain the discovery flow clearly. An LLM may not infer it should call get_tables first, then get_columns, then run_query.