AI safety and alignment research lab: probes, interventions, SAE experiments, SDK and APIs, powered by local MLX inference
mlx-dyno exposes 19 tools across two servers (lab and router). Tool definitions are present with schemas and descriptions, but quality is inconsistent. Most research operation tools (inspect, compare, probe, sae, patch_sweep) have similar parameter structures and adequate descriptions (60-80 chars), meeting baseline minimums. However, critical gaps emerge: (1) Many parameter descriptions lack actionable constraints, 'layers' says 'integers 0-255' but doesn't explain what layer indices mean or why the range matters; (2) Output schemas are completely undocumented, no tool declares what it returns, forcing LLMs to guess; (3) Error handling is absent, no guidance on retryability, user-fixable errors, or recovery paths; (4) The 'chat_completions' tool lacks parameter descriptions for 'messages' array structure and 'stream' behavior; (5) Router tools like 'set_config' accept generic 'policy' objects with no schema detail. Per-tool scores range 35 - 65, averaging 52. This is typical of community MCP servers, functional but not production-ready.
Cancel a running or queued research job
OpenAI-compatible chat completions endpoint with intelligent routing
Research operation to compare model behaviors across configurations
Download an artifact file from a completed research job
Get the current router policy configuration
Check health status of the lab service
Get details of a specific research job by ID
Get Prometheus-format metrics for the router
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields. Every tool lacks a 'returns' specification.
Parameter descriptions lack actionable constraints. 'layers' says 'integers 0-255' but doesn't explain semantic meaning (layer indices in model architecture?). 'examples' says 'max 512' but not what format or structure.
set_config accepts generic 'policy' object with no schema detail. LLMs cannot determine valid policy structure, required fields, or constraints.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 55 | 2026-07-28+ | v2 |
Get the OpenAPI specification for the lab API
Get the decision log of routing decisions made by the router
Research operation to inspect model behavior and activations
List all discovered model backends and their capabilities
List all research jobs with their status and metadata
List all available models across backends
Research operation for causal patching interventions across model layers
Research operation to probe model internals and behavior
Check health status of the router service
Research operation for Sparse Autoencoder experiments
Update the router policy configuration
No error handling guidance. Tools do not document retryability, user-fixable errors, or recovery paths. E.g., cancel_job does not explain what happens if job is already complete.
chat_completions 'messages' parameter lacks description of expected array structure (role/content fields?). 'stream' behavior undocumented, does it return Server-Sent Events or chunked JSON?