Loading ToolBench
Lagaam demonstrates solid definition quality with three well-named, verb-prefixed tools (list_, describe_, query_). All tools have substantive descriptions (100-200 chars) that explain WHAT, WHEN, and prerequisites. Input schemas are present with type declarations. However, parameter descriptions are sparse or missing entirely: query_data's 'sql' param lacks format/constraint guidance; describe_table params lack examples of valid values. Output schemas are not documented in the visible code. Error handling is strong (recovery hints, audit logging) but not surfaced in tool descriptions. The server follows pattern:tool and pattern:tool-description well but misses pattern:constrained-input and pattern:response-shaper.
Get the exact columns and types of one table. Always describe a table before querying it; column names you have not seen here are guesses.
List every catalog, schema, and table you are allowed to query. Call this first to ground yourself before describing tables or writing SQL — table names you have not seen here are guesses.
Run a read-only SELECT and get the rows back. Write a single SELECT in the engine's dialect. The query is checked for safety, priced against your budget, and executed with a row cap — so name the columns you need (no SELECT *), and add WHERE filters to keep the scan small. If it is rejected, the message says what to fix. Describe the tables first so column and table names are exact.
query_data 'sql' parameter lacks format constraints and validation guidance. Description does not specify: dialect requirements, SELECT-only enforcement, row/scan limits, or what makes a query 'safe'. LLMs cannot infer these constraints and will pass invalid SQL.
Output schemas (CatalogMetadata, TableSchema, QueryResult) are not documented in tool descriptions. LLMs cannot plan downstream calls or extract fields without knowing the response structure. Visible code imports these types but does not expose their fields.
describe_table parameters (catalog, schema, table) have minimal descriptions ('catalog name', 'schema name', 'table name'). No guidance on format, case sensitivity, or how to obtain valid values. Should reference list_catalogs as prerequisite.
Inferred effective spec: 2026-07-28+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 69 | 2026-07-28+ | v2 |
Error handling is implemented server-side (LagaamError, recovery hints) but not surfaced in tool descriptions. Descriptions do not explain what errors are possible, when to retry, or what the agent should do on failure. Recovery hints exist in code but are not discoverable by LLMs.
No pagination or result-limiting guidance in list_catalogs or describe_table descriptions. If a catalog has thousands of tables, the response could exhaust context. Descriptions should state limits and offer pagination if needed.