Loading ToolBench
FastMCP Server for Secoda - provides tools for searching data assets, documentation, running SQL queries, and retrieving entity information from Secoda
Secoda MCP has 6 tools, all read-only, with basic descriptions and properly typed parameters. However, there are significant gaps in description quality (many descriptions are too brief), output schemas are completely undocumented, and parameter descriptions lack depth and context. Tool names follow verb-noun convention but descriptions fail to explain WHEN to use each tool versus similar alternatives, or what structure is returned. No error handling guidance, no examples of output fields, and no documentation of dependencies between tools (e.g., retrieve_entity vs search_data_assets). The fastmcp framework handles schema registration, but the actual schema definitions in server.py show only input parameters with types, no output schema documentation anywhere. Descriptions range from 23 - 45 characters, below the 194-char baseline for production tools. Parameter descriptions exist but are minimal (8 - 46 chars vs. 72-char baseline).
Retrieve the lineage of an entity.
Retrieve the glossary.
Retrieve an entity from the database.
Run a SQL query on the database.
Search for data assets in the database.
Search for documentation in the database.
No output schema documentation. Tools return JSON strings with unspecified field structures. LLMs cannot plan downstream operations or extract data reliably without knowing what fields to expect.
Tool descriptions are too brief (23 - 45 chars, baseline 194 chars) and lack guidance on WHEN to use each tool. 'Search for data assets in the database' does not explain the difference from search_documentation or what use case each addresses. Prompt.py hints at the distinction, but tool descriptions should be self-contained.
Parameter descriptions are minimal and lack constraints or context. 'The search query for data assets' (27 chars) does not explain query syntax, expected format (free text vs. regex?), or what happens if results exceed limits. Missing guidance on pagination semantics, does page=1 mean 0-indexed or 1-indexed?
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 44 | 2026-07-28+ | v2 |
| 2026-03-09 | D | 51 | - | v1 |
No documented result limits or pagination behavior. search_data_assets and search_documentation accept 'page' parameter but do not specify page size, max results per page, or total count return. Returns a plain JSON string from call_tool(), structure unknown to agent.
call_tool() returns json.dumps(response.json()), a string, not structured output. Agents must parse JSON inside a string, then extract fields. No error handling; response.raise_for_status() will crash with a 500 error or connectivity issue. No recovery guidance for the agent.
No error classification or recovery hints. If API returns 404 (entity not found), 403 (permission denied), or 500 (server error), the agent receives only an exception. Should categorize errors as retryable, user-fixable, or fatal, and guide the agent's next action.
Glossary tool has no parameters but description is vague ('Retrieve the glossary'). Does it return all glossary terms? If there are thousands, this will blow the context window. Missing pagination or filtering options.
No tool composition guidance. run_sql, search_data_assets, and retrieve_entity overlap in intent (all retrieve data). Prompt.py hints at workflow (search → SQL), but tool descriptions do not explain the intended call sequence or dependencies.
API_TOKEN is passed as Bearer token in HTTP Authorization header. If the Secoda API endpoint is compromised or logs requests, the token is exposed. Code does not validate that API_TOKEN is set before use, will fail at runtime with a cryptic error.