Certification Readiness multi-agent system — a live council of AI advisors that debate whether a team can certify for Microsoft exams (SC-200, AZ-500, etc.) within a deadline. Runs on Microsoft Foundry (gpt-4.1) with grounding via Azure AI Search (Foundry IQ) and Microsoft Learn. Serves a web dashboard (quorum.html) and exposes /api/council for the orchestrated multi-agent reasoning.
DELPHAI exposes 8 tools via HTTP/Flask but lacks critical definition quality. Tool names are weak (council, learn, health, brief, download, assess, coach, burnout_assess), most do not start with action verbs and several are ambiguous. Descriptions exist but are verbose and lack LLM-optimized structure. Input schemas are present but incomplete: many parameters lack type constraints, enums are missing, and output schemas are undocumented. No error handling guidance, no parameter validation rules, and no security annotations. The server is functional but would not pass code review for production agent use.
Assessment Agent tool. Generates grounded, cited practice questions with smart distractors and runs exam-blueprint coverage gate per candidate.
Generates an HTML brief of the certification-readiness council decision with verdict, recommendation, and council record table. Returns as downloadable HTML attachment.
Burnout/Wellbeing Agent tool. Flags plans that exceed sustainable weekly capacity and can VETO for human reasons based on workload signals.
Enablement Coach tool. Provides concrete enablement plan: which weak exam domains to close, which approved/Microsoft Learn sources to study, and projected score uplift.
Runs the real Python orchestrator and returns the agents' live narration + the reconciled recommendation as JSON. Executes the multi-agent certification-readiness debate.
Tool names lack action verbs and are ambiguous. 'council', 'learn', 'brief', 'assess', 'coach', 'burnout_assess' do not follow verb_noun pattern. LLMs cannot infer intent from names alone. Should be: 'run_council', 'fetch_learn_content', 'generate_brief', 'run_assessment', 'generate_coaching_plan', 'assess_burnout'.
Input schemas lack type constraints and enums. Parameters like 'cert' accept free-form strings (e.g., 'SC-200') with no enum constraint. 'candidates' is comma-separated with no validation. 'content_type' in download has a description mentioning restrictions but no enum in schema. LLMs will hallucinate invalid values.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 39 | 2026-07-28+ | v2 |
Generic download endpoint. Accepts filename, content, and content_type; returns as attachment with sanitized filename.
Health check endpoint. Returns model mode, detail, and grounding source (foundry_iq or local).
Live exam content pulled from the public Microsoft Learn catalog. Attempts to fetch from Microsoft Learn MCP server first, falls back to REST catalog, then cached fallback.
Output schemas are completely undocumented. Tools return JSON responses (council returns model/live/turns/result; learn returns live/via/cert/title/summary/url/source; brief returns HTML attachment) but no schema is declared. LLMs cannot plan downstream calls or extract fields reliably.
No error handling guidance. Tools fail silently or return generic HTTP errors (e.g., 401 for auth, 500 for exceptions). No recovery hints, no categorization of retryable vs fatal errors, no actionable error messages. Agents cannot self-correct.
Parameter descriptions are generic or missing validation rules. 'cert' says 'Certification code (e.g., SC-200)' but does not list valid codes. 'candidates' says 'Comma-separated list' but does not specify format, length, or ID pattern. 'weeks' is a number with no min/max bounds. LLMs cannot validate inputs.