LangGraph-based agent factory workflow with FastAPI service for starting runs, resuming human approval, exposing state and build logs
12 tools with complete input schemas and descriptions, but quality is uneven. Naming follows verb_noun convention (start_run, approve_run, get_run_status, list_runs, get_build_logs, scout_researcher, pm_synthesizer, system_architect, quality_gate, human_approval, builder_agent, cancel_run). Descriptions range 40-180 chars, meeting the 10-1024 baseline. However, parameter descriptions are minimal (5-30 chars), lacking actionable constraints. No output schemas documented. Error handling absent, no recovery guidance, retryability classification, or actionable error messages. Composition is reasonable (single responsibility per tool), but chaining metadata (thread_id, project_name) is inconsistently returned. No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Risk classification present (WRITE vs READ_ONLY) but not exposed in schema.
Resume a run waiting at human approval stage with approval decision
Run Aider in Docker to implement the architecture specification
Cancel an in-flight run by thread ID
Get streaming build logs for a run in progress
Get the current status and state of a run by thread ID
Request human approval to proceed with builder execution
List all runs with optional filtering by task status, approval state, and build state
No output schemas documented. LLMs cannot infer what fields to expect from tool responses, forcing unstructured parsing and increasing hallucination risk.
Parameter descriptions are minimal (5-30 chars) and lack actionable constraints. E.g., 'The thread ID of the run' does not explain format, length, or how to obtain it. LLMs cannot validate inputs without explicit constraints.
No error handling guidance. Tools lack recovery hints, retryability classification, or actionable error messages. An LLM receiving a failure has no guidance on whether to retry, ask the user, or abort.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 56 | 2026-07-28+ | v2 |
Technical PM node: turn research into a crisp PRD-style spec with goals, users, scope, milestones, risks
Strict specification reviewer: evaluate architecture for implementability, coherence, and security awareness
Research scout node: produce concise bullet research useful for product and architecture
Start a new LangGraph workflow run with topic, project name, and max iterations before approval
System architect node: produce concrete architecture with components, interfaces, data flows, security, deployment
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Risk classification exists in metadata (WRITE vs READ_ONLY) but is not exposed in the MCP schema, preventing clients from applying safety policies.
list_runs accepts enum filters (task_status, approval_state, build_state) but lacks pagination. Default limit=500 may exceed context windows. No next_cursor or total_count documented.