Loading ToolBench
FastAPI-based server that compiles natural language workflow descriptions into Azure Logic App definitions through a multi-agent orchestration pipeline (interpret → plan → build IR → compile → critic review)
Loom has 9 tools with basic FastAPI registration but significant quality gaps. Tool names lack clear action verbs (e.g., 'interpret', 'plan', 'build_workflow_ir' are vague about what they produce). Descriptions exist but are generic and lack LLM-optimization guidance. Input schemas are present but incomplete: many parameters lack type constraints, enums, or validation hints. No output schemas documented. Error handling is minimal, tools return raw exceptions rather than actionable recovery guidance. The orchestrate tool combines multiple responsibilities (interpret→plan→build→compile→critic) violating single-responsibility principle. No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear read/write semantics.
Builds an intermediate representation (IR) of the workflow from a plan. Converts plan steps into IR steps with metadata, variables, and error handling configuration.
Compiles workflow IR into an Azure Logic App JSON definition. Generates production-aligned Logic App schema with triggers, actions, variables, and parameters.
Rule-based validation of workflow IR. Checks for required trigger, steps, and proper condition configuration (on_true/on_false branches).
Rule-based validation of compiled Logic App definition. Checks for required schema, triggers, actions, and dependency resolution between actions.
Health check endpoint. Returns server status.
Interprets natural language text into structured workflow intent. Extracts trigger type, actions, conditions, and description using an LLM-based interpreter agent.
Tool names lack action verbs and clarity. 'interpret', 'plan', 'build_workflow_ir' do not clearly signal what they do or what they return. LLMs cannot infer intent from these names alone.
No output schemas documented. Tools return complex objects (Intent, Plan, WorkflowIR, Logic App definitions) but LLMs have no visibility into the structure. Agents cannot plan downstream calls or extract required fields.
Parameter descriptions are generic and lack constraints. 'trigger' is described as 'Trigger type' with no enum of valid values (http, schedule). 'actions' is 'List of action types' with no schema for array items. LLMs cannot validate inputs.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 49 | 2026-07-28+ | v2 |
Test endpoint for LLM client functionality. Sends a prompt to the configured LLM (Ollama or Azure OpenAI) and returns the raw response.
End-to-end orchestration of the workflow compilation pipeline. Runs interpret → plan → build IR → compile → critic review sequence. Optionally deploys to Azure Logic Apps via MCP if enabled.
Converts workflow intent into a deterministic execution plan. Organizes steps in proper order (conditions first, then actions) with configuration details.
orchestrate tool violates single-responsibility principle. It runs interpret→plan→build→compile→critic in sequence, combining 5 distinct operations. Should be split so agents can compose them independently.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear semantics. orchestrate is marked WRITE but others are READ_ONLY, this metadata should be in tool definitions for agent safety.