Loading ToolBench
Incident-response MCP server for Apache Airflow with URL-first diagnosis, bounded logs, multi-instance routing, and optional recovery actions
Strong definition quality overall with well-structured tool names, comprehensive descriptions, and documented schemas. Most tools follow verb_noun naming conventions (list_*, get_*, airflow_*). All tools include detailed docstrings explaining behavior, parameters, and return structures. Parameter descriptions are thorough and include type information. Key strengths: consistent annotation of read-only vs write operations, explicit error handling documentation, pagination support on list tools. Weaknesses: some write tools (tools 12-16) have minimal descriptions visible in schema metadata; tool descriptions could be more concise (several exceed 300 chars); some tools reference optional parameters that lack clarity on when each alternative should be used (instance vs ui_url vs direct IDs).
Clear a DAG run and all its task instances (mark as success or removed).
Clear task instances within a DAG run (mark as success or removed).
Describe a configured Airflow instance (host + metadata, never secrets). Parameters - instance: Instance key (e.g., "data-stg") Returns - Response dict: { "instance", "host", "api_version", "verify_ssl", "auth_type", "request_id": str } - Raises: ToolError with compact JSON payload (`code`, `message`, `request_id`, optional `context`)
Get DAG details and a UI link. Parameters - instance | ui_url: Provide one; `ui_url` auto-resolves/validates the host. - dag_id: Required when only `instance` is supplied. Returns - Response dict: { "dag": object, "ui_url": str, "request_id": str } - Raises: ToolError with compact JSON payload (`code`, `message`, `request_id`, optional `context`)
Get a single DAG run and a UI link. Parameters - instance: Instance key (optional) - ui_url: Airflow UI URL to resolve instance/dag/dag_run (optional) - dag_id: DAG identifier - dag_run_id: DAG run identifier Returns - Response dict: { "dag_run": object, "ui_url": str, "request_id": str }
Write tools (pause_dag, unpause_dag, trigger_dag, clear_dag_run, clear_task_instances) have minimal descriptions in schema metadata, descriptions appear truncated or missing in the tool definition capture. Only tool names and parameters visible, not full docstrings.
Tools 9-11 (get_task_instance, get_task_instance_logs, list_dataset_events) have descriptions under 30 characters in the provided schema metadata ('Get a single task instance...', 'Get task instance logs...', 'List dataset events...'), violating the 10-1024 character guideline for meaningful descriptions. Real docstrings likely exist in server.py but were not captured in schema.
Mutual exclusivity between 'instance' and 'ui_url' parameters documented in descriptions, but no JSON Schema constraint (e.g., oneOf) enforces it. LLMs may pass both parameters, forcing the tool to guess precedence.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | B | 76 | <=2025-11-25 | v2 |
| 2026-03-09 | D | 53 | - | v1 |
Get a single task instance with state, duration, and attempt log URLs.
Get task instance logs for a given attempt, with log lines and metadata.
List DAG runs (defaults to execution_date DESC) with per-run UI URLs. Parameters - instance: Instance key (optional) - ui_url: Airflow UI URL to resolve instance/dag_id (optional) - dag_id: DAG identifier (required if ui_url not provided) - limit: Max results (default 100; accepts int/float/str, coerced to non-negative int, fractional values truncated) - offset: Offset for pagination (default 0; accepts int/float/str, coerced to non-negative int, fractional values truncated) - state: List of states to filter by (optional) - order_by: Optional `"start_date"`, `"end_date"`, `"execution_date"`, or `"logical_date"` (omit to use ``execution_date``; execution_date and logical_date are mapped to whichever name the target Airflow version uses) - descending: Sort direction (default True). Ignored when order_by is omitted; defaults always use execution_date descending Returns - Response dict: { "dag_runs": [{ "dag_run_id", "state", "start_date", "end_date", "ui_url" }], "count": int, "request_id": str } - Raises: ToolError with compact JSON payload (`code`, `message`, `request_id`, optional `context`)
List DAGs (pause state + UI link) for the target instance. Parameters - instance: Instance key (optional; mutually exclusive with ui_url) - ui_url: Airflow UI URL to resolve instance (optional; takes precedence - must match a configured host) - limit: Max results (default 100; accepts int/float/str, coerced to non-negative int, fractional values truncated) - offset: Offset for pagination (default 0; accepts int/float/str, coerced to non-negative int, fractional values truncated) Returns - Response dict: { "dags": [{ "dag_id", "is_paused", "ui_url" }], "count": int, "request_id": str } - Raises: ToolError with compact JSON payload (`code`, `message`, `request_id`, optional `context`)
List dataset events (Airflow 2.4+), with timestamps and associated DAG runs.
List configured Airflow instance keys. Returns - Response dict: { "instances": [str], "default_instance": str | null, "request_id": str } - Raises: ToolError with compact JSON payload (`code`, `message`, `request_id`, optional `context`)
List task instances within one DAG run, including state and attempt log URLs. Parameters - instance: Instance key (optional) - ui_url: Airflow UI URL to resolve instance/dag/dag_run (optional) - dag_id: DAG identifier - dag_run_id: DAG run identifier - limit: Max results (default 100; accepts int/float/str, coerced to non-negative int, fractional values truncated) - offset: Offset for pagination (default 0; accepts int/float/str, coerced to non-negative int, fractional values truncated) - state: Optional list of task states (case-insensitive). When provided, only matching states are returned. - task_ids: Optional list of task identifiers to include. Returns - Response dict: { "task_instances": [{ "task_id", "state", "try_number", "ui_url" }], "count": int, "total_entries"?: int, "filters"?: { "state": [...], "task_ids": [...] }, "request_id": str } - Raises: ToolError with compact JSON payload (`code`, `message`, `request_id`, optional `context`)
Pause a DAG.
Parse an Airflow UI URL, resolve instance and identifiers. Parameters - url: Airflow UI URL (http/https) Returns - Response dict: { "instance", "dag_id"?, "dag_run_id"?, "task_id"?, "try_number"?, "route", "request_id" } - Raises: ToolError with compact JSON payload (`code`, `message`, `request_id`, optional `context`)
Trigger a DAG run with optional configuration.
Unpause a DAG.
Parameters 'limit' and 'offset' accept int|float|str with coercion rules documented in descriptions ('coerced to non-negative int, fractional values truncated'). This is implicit validation, not schema-enforced. Float acceptance is unusual for pagination and invites confusion.
Destructive tools (airflow_clear_dag_run, airflow_clear_task_instances) support dry_run but do not explicitly document the recovery path if a real (non-dry-run) clear is executed by mistake. No mention of how to undo or restore cleared task instances.
Return structures for write/action tools (trigger_dag, pause_dag, unpause_dag, clear_dag_run, clear_task_instances) not documented in descriptions. LLMs cannot plan follow-up steps without knowing what these tools return.