Loading ToolBench
Thin MCP server for OrionBelt Semantic Layer — delegates to REST API
OrionBelt MCP has 9 tools with mixed quality. All tools have names starting with action verbs (load_, remove_, list_, get_, query, run_, validate_, explore_, describe_), which is positive. However, descriptions are inconsistent, some are adequate (25-60 chars), but several are too brief to fully guide LLM selection (load_model, remove_model). Input schemas are present and properly typed for all tools, but parameter descriptions are terse and lack context about formats, constraints, and dependencies. Output schemas are not documented in the visible code. Error handling is mentioned in configuration but not evident in tool definitions. The server uses fastmcp with proper HTTP transport support (via Dockerfile configuration), though the evaluated code shows STDIO as default. No tool annotations (readOnlyHint/destructiveHint) are visible despite several tools having explicit risk levels documented in the specification (load_model and remove_model are WRITE operations, others are READ_ONLY).
Describe the semantic intent of a query or model element.
Explore the schema of a loaded model to discover available tables, columns, and relationships.
Retrieve metadata and schema information about a loaded model.
List available models that can be loaded.
Load a model by name into the session and get a model_id for use in downstream tools.
Execute a natural language query against a loaded model and retrieve results.
Unload a model from the session.
Descriptions are too brief and lack context. load_model's description (38 chars) doesn't explain the semantic meaning of model_id or when to call it vs run_batch. remove_model's description (39 chars) is minimal. Pattern requires 50-200 char descriptions that guide LLM selection.
Parameter descriptions are too generic. 'The model ID to query' appears identically across multiple tools (get_model_info, query, explore_schema, describe_intent). No guidance on what a model_id looks like, how to obtain it, or its scope. Users/LLMs cannot infer the semantic type.
No output schemas documented. Tools return results but the code visible does not show return type definitions, field names, or structure. LLMs cannot plan downstream calls or extract data without knowing what fields exist. run_batch returns 'all results' but no schema shown.
Inferred effective spec: 2026-07-28+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 59 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 0 | - | v1 |
Execute a batch query that loads a model inline, executes queries, and returns all results in one call.
Validate a model configuration without loading it.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) present in tool definitions, despite risk levels being specified (load_model and remove_model are WRITE operations). Agents cannot determine which tools are safe to retry or have side effects.
Ambiguous parameter naming. run_batch accepts 'queries' (array) but no guidance on expected array item type (string? object?). query accepts 'query' (string) but does not clarify the natural language format or constraints. No examples or patterns provided.
No error handling guidance in tool descriptions. If a query fails, a model is not found, or validation fails, there is no indication of what the LLM should do next, whether to retry, or which tools to call for recovery.
Missing pagination guidance. Tools like list_models and explore_schema do not document limits, page/offset parameters, or total counts. If a semantic layer has hundreds of models or tables, result truncation will be silent.
Parameter 'element' in describe_intent lacks specificity. Description says 'table, column, or query' but does not clarify syntax, is it 'my_table.my_column', 'my_column', 'table:my_table'? No format constraint or examples provided.
API key is exposed as a configuration parameter (api_key in Settings). While the code comment notes it should use server-side secret injection, the implementation still accepts it from environment variables, risking exposure in logs or traces if the server is compromised.