Loading ToolBench
The server provides 7 tools with basic descriptions and input schemas for most tools. However, there are significant gaps in definition quality: (1) Output schemas are completely undocumented, LLMs cannot predict return types or plan downstream calls; (2) Parameter descriptions are minimal or missing context; (3) Error handling guidance is absent; (4) No pagination support documented despite tools that likely return lists; (5) Parameter constraints (ranges, enums, formats) are not declared. The tool names follow verb-noun convention (list_dags, get_dag_runs_today, trigger_dag_run, etc.), which is good. Descriptions are present but brief (50-100 chars on average) and lack WHEN/WHY context. This is a typical community-grade server with functional basics but insufficient detail for LLM reasoning.
Check for failed DAG runs in the last 24 hours. Returns a list of DAGs that had at least one failed run in the past 24 hours, with execution dates.
Check the Airflow scheduler health via the /health endpoint. Returns the scheduler status and latest heartbeat timestamp.
Get the latest run status for a specific DAG. Args: dag_id: The DAG identifier. Returns the state, execution date, and duration of the most recent run.
Get all DAG runs from today with their status. Returns every DAG run that started on the current UTC date, grouped by status (success, failed, running, queued).
Get individual task instances for a specific DAG run. Args: dag_id: The DAG identifier. run_id: The DAG run identifier. Returns the list of tasks with their state, duration, and operator.
OUTPUT SCHEMAS COMPLETELY UNDOCUMENTED. No tool documents its return type, fields, or structure. LLMs cannot predict result fields for downstream chaining, cannot validate responses, and cannot plan multi-step workflows.
NO PAGINATION DECLARED. Tools like list_dags, get_dag_runs_today, check_failed_dags, get_task_instances appear to return lists but declare no limit, offset, page, or cursor parameters. Results could be unbounded, causing context explosion.
NO ERROR HANDLING GUIDANCE. No tool describes failure cases or recovery actions. If dag_id is invalid, does the tool return 404, a descriptive error, or crash? LLM has no recovery path.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 65 | 2026-07-28+ | v2 |
List all DAGs with their paused/active status. Returns a summary of every DAG registered in Airflow, including the dag_id, whether it is paused, and whether it is active.
Trigger a manual DAG run. Args: dag_id: The DAG identifier to trigger. Returns confirmation with the new run ID and state.
PARAMETER CONSTRAINTS MISSING. Parameters like dag_id lack validation rules, format patterns, or length constraints. Input validation is server-side only; LLM cannot self-correct invalid input.
DESCRIPTIONS UNDER 100 CHARS ON AVERAGE. Tool descriptions lack WHEN/WHY context. For example, list_dags does not explain whether to use it for discovery, health checks, or initial orientation. Description 'List all DAGs with their paused/active status' is WHAT but not WHEN.
NO CHAINING METADATA. Tools like get_task_instances require dag_id and run_id but don't document where to obtain those (hint: use get_dag_runs_today to get run_id). LLM must discover dependencies by trial and error.