Loading ToolBench
An autonomous data incident responder that investigates data quality issues in a data warehouse, traces root causes through DataHub metadata and lineage, and proposes repairs to pipeline transformations.
BlackBox demonstrates solid tool design with 16 well-named, action-verb tools covering a coherent incident-response workflow. All tools have descriptions (avg ~120 chars) and input schemas with types. However, output schemas are entirely undocumented, LLMs cannot predict what fields to expect from tools like datahub_lineage or profile_column, forcing them to reason about unstructured responses. Parameter descriptions are present but often terse (e.g., 'DataHub dataset URN' lacks format guidance). Error handling is minimal, no recovery guidance, no categorization of retryable vs fatal errors. The evidence-citation pattern (evidence_ids arrays) is sophisticated but underdocumented in tool descriptions. Missing: pagination guidance for list_transforms, field-level constraints for SQL queries, and confirmation patterns for destructive repair operations.
Compare recent daily revenue / median order value / order counts against the committed healthy baseline. Shows exactly when and how much metrics diverged.
Declare the confirmed root cause. asset_urn must be the MOST UPSTREAM asset where the defect enters the pipeline (the true origin, not where it becomes visible). Requires cited evidence: at least one DataHub lineage/metadata item AND one quantitative item (profile or baseline comparison) that names the blamed field. Rejected otherwise.
Fetch full DataHub context for a dataset urn: description, schema fields with documented meanings (the data contract), ownership, tags, custom properties.
Traverse the DataHub lineage graph from an asset. Returns upstream/downstream nodes and edges (with column-level mappings where available).
Search the DataHub metadata graph for datasets/metrics by name or business meaning. Returns urns, names, descriptions.
Output schemas completely undocumented. Tools like datahub_lineage, profile_column, get_metric_history return complex nested structures (nodes/edges, per-day stats, baseline overlays) but provide zero schema guidance. LLMs cannot predict field names, types, or nesting depth, forcing them to reason about unstructured responses and increasing hallucination risk.
No error handling or recovery guidance. Tools provide no categorization of errors (retryable vs fatal), no actionable error messages, and no suggestions for recovery. E.g., run_sql could fail due to syntax, timeout, or permission, but the tool offers no guidance on what to do next.
Parameter descriptions lack actionable constraints. 'query' in datahub_search has no guidance on syntax, length limits, or expected format. 'table' in profile_column says 'schema-qualified' but doesn't specify the exact format (schema.table vs schema/table). 'new_content' in propose_repair lacks validation rules for SQL syntax.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 61 | 2026-07-28+ | v2 |
Conclude the reported symptom is NOT a data incident. Requires cited quantitative evidence showing metrics within normal range.
End the investigation with a final report (symptom → lineage → evidence → root cause → repair → verification), citing evidence ids.
Current value + 90-day daily history of the reported business metric, with healthy baseline overlay.
List the pipeline's SQL transformation files.
Per-day distribution stats (count/mean/median/min/max/null-rate) for a numeric column of a warehouse table, optionally segmented by a categorical column. Table names are schema-qualified (e.g. staging.stg_orders).
Propose the FULL new content of one pipeline transform file as the repair. The system computes the real diff, applies it, rebuilds the warehouse, reruns the entire invariant suite and the KPI. Iterate if verification fails.
Read the SQL source of one pipeline transformation.
Register a candidate explanation to investigate. target_urn = the DataHub urn of the suspected asset.
Run the pipeline's full data-quality invariant test suite against the current warehouse; returns pass/fail per test.
Run a single read-only SQL query (DuckDB) against the warehouse. Use for targeted checks the other tools don't cover.
Update a hypothesis after gathering evidence. Eliminating or confirming REQUIRES citing evidence_ids returned by fact tools.
Evidence-citation mechanism (evidence_ids arrays in confirm_root_cause, declare_no_incident, update_hypothesis) is sophisticated but underdocumented. Tool descriptions do not explain what evidence_ids are, how they are generated by fact tools, or what happens if an invalid ID is cited. This creates a hidden contract between tools.
No confirmation or dry-run pattern for destructive operations. propose_repair directly modifies pipeline files, rebuilds the warehouse, and reruns tests, but offers no dry-run, preview, or confirmation step. An LLM mistake could corrupt production data.