Loading ToolBench
DataHub change-risk agent: intent parser -> asset resolver -> impact engine -> risk engine -> recommendation engine -> write-back service
Impact Radar exposes 6 tools via FastAPI HTTP endpoints with basic parameter schemas and descriptions. However, critical gaps undermine production readiness: (1) Tool descriptions lack LLM-optimized structure, they read as API documentation rather than agent guidance (e.g., 'POST /api/change-request: Submit...' instead of 'Analyze a data asset change request...'). (2) Parameter descriptions are minimal; many lack format/constraint guidance. (3) Output schemas are not documented in tool definitions, LLMs cannot predict response structure. (4) No error handling guidance; parse errors and analysis failures lack recovery hints. (5) Tool naming mixes verbs inconsistently (change_request vs resolve_candidate vs write_back). Average tool score: 52/100.
POST /api/change-request: Submit a natural language change request describing a data asset modification (rename_column, drop_column, or deprecate_dataset). Returns either an analysis result, disambiguation candidates, or a parse error.
GET /api/assets/detail: Fetch columns and documentation for one asset by URN. Fetched lazily when a tree node is expanded.
GET /api/change-request/{analysis_id}/events: Poll the in-memory execution log for an analysis. Returns pipeline-stage lines during analysis and datahub-enrich agent's live tool-call activity during write-back. Supports incremental polling via since parameter.
POST /api/change-request/{analysis_id}/resolve-candidate: Resolve a disambiguation by selecting one candidate asset URN. Returns the full change analysis for the selected asset.
GET /api/assets: Search for datasets matching a query string, grouped by platform. Powers the left-pane tree roots in the frontend.
POST /api/change-request/{analysis_id}/write-back: Persist the planned change note to the asset's documentation in DataHub. Returns success/failure status and the applied documentation.
Tool descriptions are API-centric, not LLM-optimized. They begin with HTTP method/path (e.g., 'POST /api/change-request: Submit...') instead of agent-friendly intent language. LLMs need 'Analyze a data asset change request to identify impact' not 'POST /api/change-request: Submit a natural language change request...'
Output schemas are not documented in tool definitions. LLMs cannot predict response structure (e.g., what fields does change_request return? Is it analysis_id, candidates, or error?). This forces agents to guess and risks parsing failures.
Parameter descriptions lack format/constraint guidance. 'analysis_id' and 'urn' have minimal descriptions; no hints on format, length, or valid patterns. 'since' parameter for get_events lacks explanation of its semantics (is it a timestamp, sequence number, or offset?).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 51 | 2026-07-28+ | v2 |
Error handling lacks recovery guidance. ParseErrorOut returns a message and supported_change_types, but no hint on what to do next. When analysis fails or disambiguation is required, the tool description does not explain the agent's next step.
Tool naming is inconsistent. 'change_request' is a noun phrase (should be 'submit_change_request' or 'analyze_change'); 'resolve_candidate' mixes verb+noun awkwardly; 'write_back' is vague (write what? where?). Consistent verb_noun naming (e.g., 'submit_change_request', 'select_candidate', 'persist_change') would improve LLM clarity.