Loading ToolBench
Mission-plan-aware workflow compilation scaffold inspired by ORCHIDE
Six tools with clear, domain-specific names (validate_plan, compile_plan, render_argo, explain_policy, render_kueue, list_plans). All have descriptions (avg ~150 chars, within 10 - 1024 range). Input schemas present with type declarations and descriptions. However, output schemas are NOT documented in the visible code, LLMs cannot predict return structure. Parameter descriptions are adequate but lack format constraints (e.g., no regex for 'path', no enum for policy engines). Error handling is mentioned in descriptions (fail-closed, policy-denied) but recovery guidance is minimal. Security is well-designed (path traversal prevention via _validate_plan_path, policy bypass gated by env var), but no tool annotations (readOnlyHint, destructiveHint) are visible. Composition is sound, each tool has one responsibility, and tools chain naturally (validate → compile → render). Missing: output schema documentation, parameter format constraints, tool annotations.
Compile a plan to workflow intents. Fail-closed: a policy-denied plan yields ``status: "denied"`` with its violations and no compilation, unless ``unsafe_skip_policy=True`` (dev only).
Explain which policy rules apply to a plan and why. Always runs the authoritative OPA engine (never the baseline), so an agent can ask "why was this denied?" and get the ground truth from the Rego bundle regardless of which engine the server is configured to enforce with.
List available mission plans in the configured plan root directory.
Render Argo Workflow manifests. Fail-closed: no artifact is produced for a policy-denied plan (``status: "denied"``) unless ``unsafe_skip_policy=True``.
Render Kueue Job manifests for resource-aware scheduling. Fail-closed: no artifact is produced for a policy-denied plan unless ``unsafe_skip_policy=True``.
Report BOTH schema validity and policy admissibility for a plan. Non-blocking: it tells the caller whether the plan would pass the admission gate (``policy_allowed``) and lists any ``violations``, so an agent sees the full picture before calling ``compile_plan``/``render_argo`` (which fail closed on a denied plan by default).
Output schemas not documented. LLMs cannot predict return structure (fields, types, pagination). Tools return structured results but no schema is visible in code or descriptions.
Parameter 'path' lacks format constraints. No regex pattern, length limits, or allowed character set documented. Invites LLM to pass invalid filenames despite path validation logic in code.
Tool annotations (readOnlyHint, destructiveHint, idempotentHint) not visible. Risk/classification is documented in descriptions but not machine-readable. Agents cannot programmatically determine safety without parsing text.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 59 | 2026-07-28+ | v2 |
Error recovery guidance incomplete. Descriptions mention 'fail-closed' and 'policy-denied' but do not guide LLM on next steps (e.g., 'Call validate_plan first to see violations before compile_plan').
Parameter 'unsafe_skip_policy' is a boolean with no enum or constraint. Description says 'requires server-side environment variable' but LLM cannot know this is only honored when ORBITAL_MCP_ALLOW_POLICY_BYPASS=1.