MCP server for Power Platform Copilot Studio agent evaluation, authoring, and live chat. Provides tools to query environments, list agents, create/run evaluation test sets, update agent instructions, add knowledge sources, publish agents, export agent definitions, and conduct live conversations with agents.
Strong foundation with 14 well-named tools following verb_noun convention (list_, create_, update_, run_, get_, add_, publish_, export_, chat_). All tools have descriptions (avg 180 chars, within 10-1024 baseline). Input schemas present with type declarations and parameter descriptions. However, output schemas are not formally documented in the visible code, responses are returned as JSON strings without explicit schema definitions. Error handling uses 'ERROR: ...' convention but lacks recovery guidance and categorization. Tool annotations (ReadOnly, Destructive, Idempotent) are declared in attributes but not exposed in MCP schema. Missing pagination support on list tools and no documented rate limiting.
Adds a public-website knowledge source to the agent (the site is searched to ground answers; up to 2 path levels deep). Returns the new knowledge source id. Changes land in the draft; run publish_agent to make them live.
Adds test cases to an evaluation test set. cases is a JSON array of case objects; each case has: input (the user prompt), expectedOutput (the agent's ideal answer), expectedKeywords (array of strings the answer should contain), expectedExecutionSteps (array of tool names the agent should invoke). Returns the count added.
Has a real conversation with a published agent as the signed-in user: messages is a JSON array of 1-10 user utterances (or one plain string), sent in sequence, each waiting for the agent's reply. Returns the conversationId and the full exchange. Useful for probing behavior before/after a change and for generating real conversations (which become recorded transcripts after a delay). Works with no-auth and Microsoft-authenticated agents; agents requiring custom manual sign-in are not yet supported. On failure returns an ERROR string with a compact reason.
Creates a new evaluation test set (empty — add cases with add_test_cases). graders is a JSON array like [{"kind":"GeneralQualityGrader"},{"kind":"CompareMeaningGrader","threshold":0.6}]; see list_graders for the catalog. Returns the new testSetId.
Output schemas not formally documented. Tools return JSON strings without explicit response schema definitions, forcing LLMs to infer structure. Pattern:response-shaper requires documented return types.
List tools (list_environments, list_agents, list_graders) lack pagination parameters (limit, offset/cursor). No documented result caps. Pattern:paginated-result requires pagination for list operations.
Error handling uses 'ERROR: ...' string convention but lacks recovery guidance and error categorization. Pattern:recovery-guide requires actionable next steps (retry, ask user, fatal).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 68 | 2026-07-28+ | v2 |
Advanced/diagnostic: run a read-only Dataverse Web API GET in an environment as the signed-in user. Pass the path after /api/data/v9.2/, e.g. "botcomponents(<id>)?$select=name,data". For inspecting how test sets are stored. INTERNAL SUPPORT TOOL — not part of the public surface.
Exports the agent's complete definition as one JSON bundle — suitable as a backup before risky edits or for migration/inspection. The bundle is the agent's full internal definition and can be large.
Polls the results of a running or completed evaluation. Returns the run status (InProgress, Completed, Failed), the count of cases passed/failed per grader, and (when complete) the full case-by-case results. Useful for tracking long-running evaluations.
Lists the Copilot Studio agents in an environment, with each agent's agentId and name. Use after list_environments; the agentId is what all other tools take.
Lists the environments the signed-in user can access, with their ids and names. Use this first when the user hasn't specified an environment; present the options and let them choose. Advanced: an optional service ring can be passed — omit it to use the server's configured default.
Lists the supported grader kinds for evaluation test sets, with their configuration options. Use when composing the graders for create_test_set.
Publishes the agent's draft, making all pending changes live. Waits for the publish to complete (up to ~2 minutes) and returns the new publish timestamp.
Runs an evaluation test set against an agent: sends each case's input to the agent, collects the response, and grades it against the case's expected output and graders. Returns a runId and the grading results (pass/fail per case, per grader). Runs are asynchronous; this call returns immediately with a runId; poll get_evaluation_results to track progress.
Replaces the agent's generative instructions (the guidance that shapes all its answers). Returns the old and new text — quote the old text so the user can restore it. Changes land in the draft; run publish_agent to make them live. Some newer agent types don't support this yet and return a clear error.
Returns the signed-in identity as the evaluation service sees it, plus the live server version. Use to verify sign-in when something behaves unexpectedly, or when the user asks what version is running.
Tool annotations (ReadOnly, Destructive, Idempotent) declared in C# attributes but not exposed in MCP tool schema. LLMs cannot see these hints to plan safely.
dv_query marked as INTERNAL SUPPORT TOOL but exposed as a public tool. Diagnostic/internal tools should be gated or documented as unstable.