Loading ToolBench
Model Context Protocol (MCP) server for Apache Airflow API integration. Provides comprehensive tools for managing Airflow clusters including service operations, configuration management, status monitoring, and request tracking.
The server defines 8 tools with basic parameter schemas and descriptions, but multiple critical quality gaps prevent a higher score. Tool naming is acceptable (verb-noun pattern), but descriptions are inconsistent in depth and clarity. Parameter documentation is present but lacks critical constraints (ranges, enums, format specifications). Output schemas are not formally documented, the code shows return types via examples but not structured schema declarations that LLMs can parse. Error handling is minimal; functions return strings or dicts but lack actionable error guidance. The [Tool Role] prefix in descriptions is non-standard and wastes tokens. Three tools (list_assets, list_asset_events) are documented as '[V2 New]' but the implementation details suggest they are placeholders or incomplete. Overall, the definitions reflect a functional baseline but fall short of production-grade quality expected for agent integration.
[Tool Role]: Lists all recently failed DAG runs in the Airflow cluster.
[Tool Role]: Retrieves detailed information for a specific DAG.
[Tool Role]: Retrieves detailed information for multiple DAGs in batch with latest run information.
[Tool Role]: Provides comprehensive prompt template for LLM interactions with Airflow operations.
[V2 New] List asset events for data lineage tracking. Asset events track when assets are created or updated by DAGs. This enables data lineage tracking and data-aware scheduling in Airflow 3.0.
[V2 New] List all assets in the system for data-aware scheduling. Assets are a key feature in Airflow 3.0 for data-aware scheduling. They enable workflows to be triggered by data changes rather than time schedules.
Tool descriptions include non-standard '[Tool Role]:' prefix that wastes tokens and does not follow LLM-optimized format. Descriptions should be 50-200 chars and directly state what the tool does, not metadata labels.
Numeric parameters (limit, offset) lack explicit range constraints. Code shows defaults (limit=20, offset=0) but no documented min/max bounds. LLMs may pass unbounded values causing API stress or invalid requests.
Output schemas are not formally documented. Return types are inferred from code (e.g., Dict[str, Any]) but LLMs have no structured schema showing which fields are present, their types, and whether they are nullable. Pattern requires documented output schemas.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-21 | D | 52 | <=2025-11-25 | v2 |
| 2026-03-09 | C | 66 | - | v1 |
[Tool Role]: Lists all DAGs registered in the Airflow cluster with pagination support.
[Tool Role]: Lists all currently running DAG runs in the Airflow cluster.
Error handling returns unstructured strings or bare exceptions without recovery guidance. Code catches exceptions generically (except Exception) and appends to an errors list, but does not categorize as retryable, user-fixable, or fatal. LLMs receive no actionable next steps.
Filter parameters (id_contains, name_contains, uri_pattern, asset_uri, source_dag_id) lack format or pattern constraints. Descriptions say 'optional' but do not specify case sensitivity, regex support, or matching semantics. LLMs must guess whether 'proj' matches 'project-name' or requires exact substring.
running_dags and failed_dags return no parameters but provide minimal discovery. No way to filter by DAG ID, time range, or status. Descriptions are too brief (under 50 chars stripped of prefix) to guide LLM selection. When to choose failed_dags over running_dags is implied, not explicit.
list_assets and list_asset_events are marked '[V2 New]' in descriptions but code is not visible; registration details and full parameter documentation cannot be verified. If these are incomplete or placeholder tools, they should not be exposed until ready.
No tool declares permission requirements (e.g., 'requires read:dags', 'requires write:airflow'). Agents have no visibility into what scopes they need or whether calls will fail due to insufficient permissions. Audit trail is also not visible in provided code.
All tools are READ_ONLY risk classification, which is correct for safety, but no confirmation or dry-run mechanism exists for compound operations (e.g., get_dags_detailed_batch processes multiple DAGs with partial failure handling but cannot preview results before full execution).