Loading ToolBench
Autonomous ML platform reliability agent that detects data quality issues (target leakage, blast radius, stale training data, missing ownership) in ML models via DataHub's metadata graph, explains findings via LLM, investigates with agentic tool-calling, writes evidence back to DataHub, and generates remediation PRs.
Server has 9 tools with reasonable naming (verb-noun pattern) and descriptions present for all. However, parameter schemas lack type definitions for several critical fields (e.g., 'urns' in get_entities is 'string or list[string]', not valid JSON Schema), and descriptions are inconsistent in depth. Some tools like search and get_lineage have good parameter documentation, but others like add_owners lack clarity on ownership_type enum values. Output schemas are not documented. Error handling and recovery guidance are absent from descriptions. Baseline: 194 chars avg description length, most tools here are 80-150 chars, below median. No tool reaches A-grade quality; most are solid C/B range.
Assign ownership to DataHub entities. Used to mark data ownership for governance.
Apply one or more tags to DataHub entities (e.g., 'model-at-risk', 'leakage-suspect'). Used to mark findings on affected models and datasets.
Apply glossary terms to DataHub entities. Used during semantic triage to mark datasets with governance terms like 'PostOutcomeEvent'.
Retrieve full metadata for one or more DataHub entities by URN, including properties, glossary terms, ownership, and custom properties
Traverse DataHub's metadata lineage graph upstream or downstream from an entity, returning all dependent/dependency entities with their metadata
Create and save an Analysis document in DataHub linked to affected entities. Used to store incident memos with evidence, investigation traces, and remediation guidance.
Parameter type definitions are invalid JSON Schema. 'urns' in get_entities declared as 'string or list[string]', JSON Schema requires oneOf or union syntax. 'tag_urns' and 'term_urns' lack enum constraints despite being URN references with known patterns.
Output schemas are not documented. Tools return complex nested structures (e.g., search returns 'searchResults' array with 'entity.info.title') but LLMs have no schema to parse them. Baseline: 100% of A+ tools document return types.
Error handling and recovery guidance absent. No tool description explains what to do if a URN is invalid, a search returns no results, or a write-back fails. Baseline: error responses must tell LLM what to do next.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 56 | 2026-07-28+ | v2 |
Keyword search across DataHub entities with optional filtering by entity type (dataset, mlModel, mlFeature)
Search for DataHub Analysis documents by title or content, used to check for idempotent re-application of findings
Update or append to an entity's description in DataHub. Used to document findings and remediation guidance on affected models.
Destructive tools (add_tags, add_terms, update_description, add_owners, save_document) lack confirmation or dry-run support. No tool description warns that these are irreversible or suggests idempotency checks. Code shows _already_applied() logic but tool descriptions do not mention it.
Parameter descriptions lack actionable constraints. 'num_results' in search has no min/max bounds (baseline: 1 - 100 range expected). 'max_hops' in get_lineage has default 3 but no explanation of cost/performance implications. 'operation' in update_description should be an enum but is free-form string.