Portable Score Studio MCP server and agent plugin packages for managing datasets, models, evaluations, deployments, and workflows
Strong schema definitions with Zod validation and consistent tool registration. All 12 tools have descriptions (avg 95 chars) and input schemas with type constraints. Tool names follow verb_noun pattern (list_, get_, start_). Annotations (readOnlyHint/destructiveHint) present for all tools. Key gaps: output schemas not documented in code; parameter descriptions are minimal (avg 40 chars, below 72-char baseline); no error recovery guidance; no pagination metadata (total_count, next_cursor) in list tools despite accepting limit/offset.
Get a dataset and its version lineage by organization and slug.
Get metrics, slices, intervals, and evidence for an evaluation run.
List versioned datasets in a Score Studio organization.
List deployments, revisions, and runtime state in an organization.
List measured evaluation runs and their current states.
List models and versions in a Score Studio organization.
Output schemas not documented. LLMs cannot infer what fields list_datasets, get_dataset, or start_training return, forcing them to guess downstream field names for chaining calls.
List tools (list_datasets, list_models, list_evaluations, list_deployments) accept limit/offset but response structure not documented. No indication of total_count or next_cursor for pagination awareness.
Error handling returns raw API errors (status, code, message, details) without recovery guidance. LLM cannot determine if error is retryable, user-fixable, or fatal.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 63 | 2025-06-18+ | v2 |
List typed Score Studio workflow definitions and status.
Run a workflow against an existing Score Studio image object. The result may report deferred blocks.
Create an immutable measured evaluation run for a model version and dataset version.
Start a tracked training job against an immutable dataset version. This may consume compute credits.
Run and retain deployment health verification evidence without changing the active revision.
Return the authenticated Score Studio user and organization memberships.
Parameter descriptions are minimal (avg 40 chars). E.g., 'Organization slug' lacks context on format, length, or examples. 'Compute provider' enum values (local, runpod, lium) not explained.
Destructive/write tools (start_training, start_evaluation, verify_deployment, run_workflow) lack dry-run or confirmation step. Agents cannot preview consequences before committing compute credits or state changes.