Loading ToolBench
Model Context Protocol (MCP) and REST interface for PyMDP active inference
PyMDP MCP has significant definition quality gaps. While 13 tools are present with basic descriptions and some parameter documentation, most tools suffer from incomplete schemas, missing parameter descriptions, and vague naming that doesn't align with Arcade patterns. Tool descriptions are present but often generic (e.g., 'Retrieve an agent by name' for get_agent). Parameters are documented with descriptions, but many lack explicit type constraints, enum definitions, or format specifications. No documented output schemas are visible in the source code. Error handling guidance is absent. The server relies on JSON string parameters for complex inputs (A_dims, B_dims, generative_model) rather than structured types, which forces LLMs to serialize/deserialize and risks malformed payloads. Schema completeness is inconsistent, some parameters specify type and description, but critical fields like observation (which takes JSON string) lack format validation or constraint documentation.
Create an active inference agent with specified parameters.
Create a simple grid world environment.
Define random A and B matrices based on dimensions.
Retrieve an agent by name.
Get all stored agents.
Get all stored environments.
Retrieve an environment by name.
No documented output schemas for any tool. LLMs cannot plan downstream calls or extract needed data without knowing what fields to expect. Critical for agent composition.
JSON string parameters (observation, generative_model, grid_size, reward_locations, action) lack format specifications, examples, or validation rules. LLMs frequently misformat JSON serialization, causing silent failures. Tools should accept structured objects or provide strict format documentation and examples.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 47 | 2026-07-28+ | v2 |
Optimize policies based on expected free energy.
Infer hidden states given an observation.
Run a simulation for a specified number of timesteps.
Sample an action from the agent's policy distribution.
Update the environment given an agent's action.
Generate a visualization of a simulation.
Parameter descriptions are missing or trivial for critical fields. Examples: observation ('JSON string of observation indices') lacks format spec; method lists valid enum values in description instead of schema enum constraint; planning_horizon has no bounds. LLMs cannot enforce constraints they cannot parse.
No error recovery guidance. Tools lack descriptions of failure modes, recovery steps, or next actions. Example: if create_agent fails with invalid generative_model JSON, what should the agent do? Call define_generative_model? Retry?
Tool composition risks: create_agent expects generative_model parameter but define_generative_model only documents A and B matrices, is C (preferences) required? Is D (prior)? get_agent returns agent object but structure is undocumented, forcing agent to guess what infer_states expects.
Tool descriptions lack dependency and sequencing hints. Agents don't know: must create_grid_world_env before step_environment? Can infer_policies be called before infer_states? Undocumented prerequisites force trial-and-error execution.
Ambiguous tool names and overlapping responsibility: get_agent (single), get_all_agents (list) are clear, but define_generative_model vs create_agent overlap in model creation. Should define_generative_model just return the model, or does it persist? Unclear ownership.
Numeric parameters lack bounds. planning_horizon, num_timesteps have no min/max documented. Agents can pass absurd values (0, negative, millions) that may hang or fail ungracefully.
Trivial tool descriptions (<50 chars). get_agent, get_environment, get_all_agents, get_all_environments lack context on when to use them vs creating new objects. LLMs cannot distinguish retrieval from creation intent without richer descriptions.