Loading ToolBench
MCP server providing tools for Soar cognitive architecture development, including datamap manipulation, project validation, layout management, and agent runtime control
28 tools with complete schemas and descriptions, but significant quality gaps. Tool names follow verb_noun convention (datamap_get_tree, agent_runtime_connect) which is good. However, descriptions are terse (avg ~60 chars, well below the 194-char baseline for A+ tools) and lack context on WHEN to use each tool or prerequisites. Parameter descriptions are sparse or missing entirely. No output schemas documented. Error handling guidance absent. Many tools expose implementation details (SML socket, vertex IDs) rather than abstracting to user-friendly concepts. Risk classifications present but no recovery guidance.
Execute Soar 'epmem' CLI command to manage episodic memory
Execute Soar 'explain' CLI command to get explanation of operator selection
Execute Soar 'explain' CLI command to track operator selection
Execute Soar 'explain' CLI command to stop tracking operator selection
Execute Soar 'preferences' CLI command to inspect preferences
Execute Soar 'print' CLI command to inspect working memory
Execute Soar 'production' CLI command to manage productions
No output schemas documented. LLMs cannot plan downstream calls or extract required fields. All 28 tools lack return type documentation.
Descriptions are terse (avg ~60 chars vs 194-char baseline). Most lack WHEN to use, prerequisites, or state-change implications. E.g., 'Execute Soar CLI command' does not explain when to call it vs agent_runtime_exec_cli.
Parameter descriptions missing or minimal. 'agent' param appears in 15+ tools with no description of what it means or how to obtain it. 'parentVertexId' and 'targetVertexId' expose internal graph structure without explaining user-facing semantics.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 54 | <=2025-11-25 | v2 |
Connect to a running Soar kernel over SML socket for agent runtime control
Disconnect from the Soar SML socket runtime session
Execute a Soar CLI command and return the output
Get current agent runtime connection/session state. Requires agent_runtime_connect before.
List Soar agents currently available in the connected runtime
Pause agent execution
Run decision cycles for an agent (or continue running)
Step an agent by one or more single decision cycles
Check the datamap for structural integrity problems: dangling linked attributes (edge target missing from the datamap) and linked attributes whose target is not reachable from the datamap root
Create a new attribute and target vertex under a SOAR_ID parent
Create a linked attribute edge to an existing SOAR_ID vertex
Delete attribute edge, and optionally remove only link edge
Get datamap tree for a Soar project file
Update attribute name/comment, and enumeration values when applicable
Add a folder node/directory under a layout parent node
Add an impasse operator node/file under a layout parent node
Add an operator node/file under a layout parent node and update datamap
Add a production file node/file under a layout parent node
Find layout nodes by node ID or by name (case-insensitive substring match). Returns resolved file paths, parent node ID, and datamap ID. Useful for discovering node IDs before calling other layout tools.
Get the currently active Soar project from workspace state persisted by the extension
Validate all project Soar files against the project datamap. Also reports datamap structural integrity issues and stale/unused datamap items (attributes never tested or created by any production).
No error handling guidance. Tools declare WRITE/DESTRUCTIVE risk but provide no recovery hints. E.g., datamap_delete_attribute_edge offers no guidance on what to do if deletion fails or how to undo.
Composition gaps: agent_runtime_cli_* tools (production, print, preferences, epmem, explain_*) are thin wrappers around Soar CLI. No batch variants. Agents calling these in loops waste tokens. Consider wrapping common sequences (e.g., track + explain operator) into single tools.