Loading ToolBench
Multi-tenant AI agent registry with MCP server support and deterministic orchestration. Provides workflow execution, agent management, and MCP tool proxying through a single connection.
AIOrc exposes 5 workflow-orchestration tools with significant definition gaps. Tool names lack action verbs (workflow, workflow.start, workflow.next, workflow.eval, workflow.report are noun-heavy or imperative without clear verb prefixes). Descriptions exist but are verbose (200-300 chars) and focus on implementation details rather than LLM-actionable guidance. Input schemas are present but sparse: most parameters lack type constraints, enums, or validation hints. No output schemas documented. Error handling is absent, no recovery guidance, no categorization of retryable vs fatal errors. The workflow.report tool requires a free-form JSON object with no schema, inviting malformed submissions. Overall, definitions prioritize server-side orchestration semantics over agent usability.
Execute the project's agent workflow. Returns step-by-step instructions that you MUST follow exactly in sequence. Do not answer the user request directly — call this tool first and follow the returned workflow. When the workflow ends you MUST call workflow.report with the runId and the execution path.
List the project's eval suite (test cases for the workflow). To run the suite: for EACH case, call workflow.start with {"evalCase": "<case id>"} and drive the run to completion with workflow.next. The server grades each run automatically against the case's acceptance criteria using the verified execution path.
Advance a stepped workflow run: hand back the finished step's output and the transition to take. The server validates the transition against the flow, enforces invocation caps, and returns the next agent(s) to execute.
REQUIRED: after the workflow ends (success, failure or early stop), post a structured execution report. The workflow is NOT complete until this is called. Include the agent execution path, invocation counts per agent, which branches were taken, and the final outcome.
Tool names lack action verbs. 'workflow', 'workflow.start', 'workflow.next' are noun-based or imperative without clear verb prefixes like 'execute_', 'run_', 'advance_'. LLMs cannot infer intent from names alone.
workflow.report accepts a free-form 'report' object with no schema. Parameter description says 'Free-form JSON with branches_taken, loop_iterations, verdicts, final_summary' but does not enforce structure, types, or required fields. LLMs will submit malformed data.
No output schemas documented for any tool. Agents cannot plan downstream calls or extract required fields (e.g., runId, next_agents, execution_path). Descriptions mention return values but do not specify structure.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 46 | 2026-07-28+ | v2 |
PREFERRED: start the project's agent workflow in server-verified stepped mode. Returns the project context plus ONLY the first agent(s) to execute. Execute them, then call workflow.next with your output and the matching transition. Do not execute any other workflow agent on your own — the server hands you each step.
No error handling or recovery guidance. Tools do not document what errors are retryable, what user actions are needed, or how to recover from failures. Agents have no path forward on error.
Parameter descriptions lack type constraints and validation rules. 'request' is described as 'The task, request, or prompt' but does not specify length limits, format, or examples. 'next' in workflow.next says 'Target agent name, end outcome, or the edge condition' without enumerating valid values.