Loading ToolBench
Daily regulatory monitoring agent for EU financial entities. Integrates Gemini 3, Fivetran MCP, and BigQuery to monitor regulatory changes, score compliance impact, and track remediation tasks.
RegPipeline exposes 6 tools with severe definition gaps. Tool names lack consistent verb-noun structure (fivetran-list-connections vs fivetran-modify-connection-state use hyphens inconsistently). Descriptions exist but are minimal (10-80 chars, below the 50-200 char LLM-optimized baseline). Most critically: tools 4, 5, 6 (bigquery.insert(audit_log), notify.send_digest, flag.query_patches) are INFERRED from agent.js comments, not explicitly registered in visible code, their schemas and actual implementations cannot be verified. Tool 1 (fivetran-list-connections) has empty input schema {}. Tool 3 (bigquery.insert(obligations)) has a complex nested array schema but no parameter descriptions. No tool includes error handling guidance, recovery hints, or output schema documentation. No tool annotations (readOnlyHint, destructiveHint, idempotentHint) present.
Record audit log entries for approval-gated actions (resync, digest send, task save)
Insert remediation tasks into the shared Obligation ledger in BigQuery
List all Fivetran connectors across all groups with health status (sync state, setup state, schema changes, last sync time)
Resume a paused or delayed Fivetran connector to restart its sync
Flag schema changes requiring manual intervention (implied tool from proposedAction)
Send the daily regulatory digest to stakeholders (implied tool from proposedAction)
Tools 4, 5, 6 are inferred from comments, not explicitly registered. No visible tool definition code, schemas, or implementation. Cannot verify actual behavior.
Tool 1 (fivetran-list-connections) has empty input schema {}. No parameters documented. LLM cannot infer what inputs are accepted.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields (e.g., connectionId for chaining).
No error handling guidance. Tools lack recovery hints (e.g., 'If connector not found, call fivetran-list-connections first'). No categorization of retryable vs fatal errors.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 40 | 2026-07-28+ | v2 |
Naming inconsistency: fivetran tools use hyphens (fivetran-list-connections), bigquery tools use dot notation (bigquery.insert(obligations)). No consistent verb-noun pattern. LLM disambiguation suffers.