Loading ToolBench
Local-first decision and scoped temporal memory for AI agents
ShadowGraph presents a comprehensive decision-graph and memory system with 30 tools. Strengths: nearly all tools have descriptions (194-char baseline), well-defined enums for status fields, and consistent parameter naming conventions. Critical weaknesses: (1) Output schemas are completely undocumented, no response fields are specified anywhere in the tool definitions, forcing LLMs to guess what each tool returns and breaking downstream tool chaining; (2) parameter descriptions exist but are often terse (under 50 chars) and lack usage guidance; (3) multiple tools operate on overlapping concerns without clear disambiguation (e.g., shadowgraph_search vs shadowgraph_retrieve vs shadowgraph_recall all retrieve information but differ subtly); (4) no visible error handling guidance or recovery patterns; (5) idempotency is addressed via idempotencyKey params but not consistently across all write tools. The schema structure itself is sound (proper JSON Schema types, min/max constraints), but the absence of documented output shapes is a critical gap for LLM composition.
Acknowledge that a review signal has been addressed. Marks signal as resolved.
Record an attempt to execute a decision or action. Tracks execution history and outcomes.
Add evidence to influence confidence in a decision or fact. Supports multiple evidence types and source classes.
Create a new decision record with full metadata. Alternative to shadowgraph_record_decision with extended schema.
Record a fact with optional temporal validity. Facts represent stable claims about the world, used for decision context.
Apply a bulk memory operation plan. Handles multiple memory operations (ADD, UPDATE, DELETE) atomically.
Build contextual information for decision-making. Aggregates relevant decisions, memories, and facts.
Output schemas completely undocumented. No tool specifies what fields are returned or their types. LLMs cannot compose tools reliably when they don't know what data they receive.
Multiple retrieval tools with unclear distinctions. shadowgraph_search, shadowgraph_retrieve, and shadowgraph_recall all retrieve information but descriptions do not clearly explain when to use each. Ambiguity causes LLM selection errors.
No error handling or recovery guidance documented. Tools return generic descriptions without mentioning what errors they can raise, how LLMs should respond, or what to try next.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 52 | 2025-06-18+ | v2 |
Export all records in the database as structured data. Used for backup, analysis, or migration.
Retrieve the change journal for audit and debugging. Returns timestamped record of all mutations.
Get pending review requests and review signals. Shows what needs human or agent attention.
Create relationships between decisions, such as 'depends_on', 'contradicts', 'similar_to', 'informs'.
Maintain memories with temporal validity. Updates or extends memory records without creating duplicates.
Get a summary of a project for preview before purge. Shows what would be deleted.
Delete all records in a project. Supports logical (default) or hard physical deletion.
Rebuild internal indexes and structures. Used for maintenance and recovery from corruption.
Semantic recall of memories and facts. Uses embeddings for similarity-based retrieval.
Request reconsideration of a closed decision. Opens it for re-evaluation with new context.
Record a decision with context, options considered, and reasoning. Creates or updates a decision record in the graph.
Redact sensitive information from records. Performs logical deletion while preserving audit trail.
Record a memory with temporal scope and identity. Stores user preferences, profiles, goals, instructions, procedures, episodes, or general notes.
Retrieve specific decision, memory, or fact records by semantic or vector similarity. Returns full record details with confidence and provenance.
Request review of decisions or memories. Creates review signals for human or agent inspection.
Search for decisions, memories, and facts by keyword or phrase. Returns matching records with relevance scoring and brief context.
Record the actual outcome of a decision after execution. Updates decision status and captures results.
Get statistics about the decision graph. Returns counts of decisions, memories, facts, and other metrics.
Mark a decision as superseded by another. Preserves history and tracks decision evolution.
Walk the decision graph from a starting point, following relationships. Returns decision paths and connected context.
Update an existing decision record with new status, outcome, or evidence. Preserves history and tracks changes.
Change the status of a decision (open, closed, pending, reopened, superseded).
Validate the integrity of the decision graph. Checks for consistency and reports any issues.
Destructive tools (shadowgraph_purge_project, shadowgraph_redact) lack confirmation or dry-run patterns. No guidance on how LLMs should safely test before committing.
Parameter descriptions are terse (under 50 chars on average) and lack usage examples or constraints. E.g., 'Filter by signal status' does not explain what valid statuses are.
Idempotency is inconsistent. shadowgraph_record_decision and shadowgraph_remember support idempotencyKey, but shadowgraph_link_decisions, shadowgraph_update_decision, and others do not. LLMs cannot safely retry writes without idempotency guarantees.
Tools returning lists (shadowgraph_search, shadowgraph_recall, shadowgraph_get_journal) document limit and offset but do not specify total_count or next_cursor. LLMs cannot determine if there are more results.
No documented scope or permission declarations. Tools accept sourceClass enum (agent_claimed, tool_observed, human_confirmed, production_verified) but no security descriptions explain what permissions are required or enforced.