Loading ToolBench
MCP server for Bauplan, an AI-first data lakehouse with Git-for-data branching and serverless multi-language pipelines
Server has 31 tools with mixed quality. Strengths: tool annotations (destructiveHint/readOnlyHint) are present and correctly applied; descriptions exist for most tools with detail on when/why to use them; parameters are mostly typed with descriptions. Weaknesses: 8 tools (delete_branch, delete_namespace, delete_table, delete_tag, import_data, merge_branch, plan_table_creation, revert_table) lack visible descriptions in provided code, descriptions may exist in source files not shown, but cannot be scored if not provided; parameter descriptions are sometimes generic ('Optional ...'); output schemas not documented; error handling lacks recovery guidance. Naming is generally strong (verb_noun pattern: create_, delete_, get_, run_). Composition is good, tools are single-purpose and chain well (plan_table_creation → apply_table_creation_plan). Parameter naming uses suffixes (_id, _branch, _namespace, _ref) to disambiguate.
Apply a table creation plan produced by plan_table_creation. Use this after reviewing or editing a plan, especially when schema conflicts require manual resolution.
Cancel a Bauplan job by ID. Use this only when the user explicitly wants to stop a running execution or long operation.
Run a Bauplan project from source files provided directly by the client. Use this for remote workflows where project_run cannot access client-local files.
Create a writable, zero-copy development branch from an existing branch, tag, or commit ref. Use this before catalog changes so writes happen in isolation before review and merge.
Create a namespace on a writable Bauplan branch. Use this before creating or importing tables into a namespace that does not exist yet.
23 of 31 tools lack visible descriptions and schemas in provided code. Descriptions appear to be in separate files (mcp_bauplan/tools/*.py) not included in the evaluation sample. Cannot assess tool-description quality for most tools.
Output schemas not documented for any tools. LLMs cannot plan downstream tool calls or extract data without knowing what fields/types to expect. Missing return type documentation.
Inferred effective spec: 2026-07-28+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-21 | F | 48 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 35 | - | v1 |
Create an Iceberg table from files matched by an S3 URI. Use this when the user wants Bauplan to infer a schema and create the table before importing data.
Create a stable catalog tag pointing to a branch, tag, or commit ref. Use this to mark a known catalog snapshot for later inspection or reuse.
Error handling lacks recovery guidance. No indication of retryability, user-fixable conditions, or next steps when tools fail. Error messages should guide LLM to appropriate recovery action.
Destructive tools (delete_branch, delete_table, apply_table_creation_plan, code_run, project_run) lack dry-run or confirmation steps. No confirmation-request pattern, agents cannot verify intent before irreversible operations.
Parameter descriptions sometimes generic ('Optional ...', 'Job priority'). Should include format, range, constraints, and rationale. E.g., 'client_timeout' lacks guidance on recommended values for different operation types.
Pagination not evident in tools returning lists (get_branches, get_commits, get_tables, get_jobs, etc.). No limit, offset, page_size, or next_cursor parameters visible. Large result sets risk context window exhaustion.