Loading ToolBench
A multi-stage reconciliation break triage and remediation system with approval-gated write operations, combining deterministic evidence gathering, LLM-based diagnosis, and human-in-the-loop action planning.
Recon Sentinel has 11 tools with complete input schemas and descriptions. Naming follows verb_noun convention (get_*, search_*, mark_*, rerun_*, flag_*). However, descriptions vary significantly in quality, some are excellent (get_txn_record, rerun_job explain context and patterns), while others are terse (get_batch_status, get_traces lack actionable guidance). Parameter descriptions are present but minimal. Output schemas are not documented. Error handling is absent, no guidance on retryability, user-fixable errors, or recovery paths. The write-gated tools (mark_resolved, rerun_job, flag_for_review) accept an ApprovalToken object but do not document token validation, expiration, or failure modes. Search tools (search_runbooks, find_similar_past_breaks) acknowledge a TODO for embedding-based upgrades but lack pagination guidance. Overall: solid foundation, but descriptions need LLM-optimization and error handling is missing.
Nearest-neighbor search over episodic memory of resolved breaks — used both for TRIAGE's benign-pattern check and to enrich DIAGNOSE. Delegates to memory/episodic.py's search_similar, which uses the same keyword-overlap approach as search_runbooks above (see that function's docstring for the embedding-based upgrade path). Returns an empty list until reflect_node has actually persisted at least one outcome — that's expected on a fresh database, not an error.
Escalates to a human review queue — appropriate for patterns (schema_break, duplicate_submission) where the correct fix requires engineering judgment or a reversal decision this system should never make autonomously, even with approval to *flag* it.
Fetch batch job run statuses in a time window (for timing-mismatch diagnosis, e.g. 'settlement batch ran before gateway batch completed').
Fetch deploy/schema-change events in a window — a huge share of reconciliation breaks trace back to one of these (see the schema_break pattern in data/synthetic/generate.py).
Fetch log lines for a service in a time window, optionally filtered by substring match against the log message.
Output schemas not documented. LLMs cannot plan downstream calls or extract required fields (e.g., what fields does get_txn_record return? Is txn_ref included for chaining?). Baseline: 100% of A+ tools document return types.
Error handling absent. No guidance on retryability, user-fixable errors, or recovery paths. E.g., get_txn_record returns None if record missing, is this expected? Should the LLM retry or try a different system? Baseline: pattern:recovery-guide requires actionable error messages.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 57 | 2026-07-28+ | v2 |
Return newly detected reconciliation breaks since a timestamp. Reads from break_feed (populated by seeded incidents in data/synthetic/generate.py). In a real deployment this table would be fed by an actual break-detection job; here it's pre-seeded.
Fetch distributed trace spans correlated to a transaction reference. Note: the generator (Stage 1) doesn't currently seed the `traces` table with data — it's schema-ready but empty. Extend data/synthetic/generate.py if a break pattern needs trace evidence.
Fetch a single transaction record from core_ledger / payment_gateway / settlement (read-only synthetic tables). system must be one of: core_ledger, payment_gateway, settlement. Returns None if no record exists in that system for that txn_ref — a missing record IS evidence (e.g. the timing-mismatch pattern).
Records the break as resolved without further data mutation — appropriate for patterns like currency_rounding_mismatch where the drift is within tolerance and no correction is actually needed, just a documented sign-off.
Simulates re-running the settlement batch for one transaction — the correct remediation for timing_mismatch, where the settlement batch ran before the upstream feed had settled and simply needs to process the transaction it originally missed. Concretely: copies the core_ledger record for txn_ref into settlement if it's not already there, exactly what a real batch re-run would produce for a transaction that only missed its window.
Keyword-overlap search over the runbook corpus, ranked by Jaccard similarity between query tokens and (title + text) tokens. TODO (upgrade path): replace with embedding similarity — embed runbooks.json once at startup into a FAISS index, embed the query, cosine-rank. Keeping the tool signature identical means nothing calling this tool needs to change when you make that swap.
ApprovalToken parameter (mark_resolved, rerun_job, flag_for_review) lacks schema detail. Token is described as 'object with token_id, break_id, action_type, expires_at, approved_by, used fields' but no JSON Schema type/properties defined. LLMs cannot validate token structure before calling.
Search tools (search_runbooks, find_similar_past_breaks) lack pagination guidance. No mention of result limits, next_cursor, or total_count. Baseline: pattern:paginated-result requires pagination for list-returning tools.
Descriptions for get_batch_status and get_traces are terse (50 - 60 chars). Baseline: 10 - 1024 chars, average 194 chars. These lack context on WHEN to call them and what patterns they diagnose.