Loading ToolBench
Production LLMOps pipeline: MCP + LangGraph + LiteLLM + Instructor + Guardrails. Processes inbound sales emails through a multi-agent workflow with CRM enrichment, RAG-based brief generation, and human-in-the-loop decision points.
Orion exhibits significant definition quality gaps across all 9 tools. While tool names follow verb_noun convention (query_contact, send_email, create_page), descriptions are present but generic and lack LLM-optimization guidance. Input schemas are visible and typed, but parameter descriptions are minimal (avg ~40 chars vs baseline 72). No output schemas documented. Error handling is absent, tools return demo data or raw API responses with no recovery guidance. No tool annotations (readOnlyHint/destructiveHint) despite clear risk classifications. Composition is reasonable (single responsibility per tool), but missing idempotency guarantees and chaining metadata. Security: credentials injected server-side (good), but no audit logging visible in code.
Compute deal score based on intent, urgency, and memory boost. Returns score (0-10), similar closed deals count, and average days to close.
Create a new Notion page with title, content, and properties.
Post message to Slack channel with optional mention.
Query CRM for contact information by email address. Returns contact ID, existence status, tier level, and past revenue.
Read email thread by email ID. Returns email metadata including sender, subject, and body content.
Search Qdrant vector database for similar case studies. Returns top-k results with similarity scores.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields (e.g., does query_contact return contact_id or id?). Baseline: 100% of A+ tools document return types.
Parameter descriptions are minimal (avg ~40 chars vs baseline 72). E.g., 'intent' in compute_deal_score lists enum values but lacks guidance on when to use each. 'memory_boost' lacks range/constraint info. Baseline: 100% of A+ tools have full param descriptions.
No error handling or recovery guidance. Demo mode returns synthetic data; production mode returns raw httpx responses. No actionable error messages (e.g., 'User not found. Try search_users() first'). Agents cannot self-correct on failures.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 54 | <=2025-11-25 | v2 |
Send email to recipient with subject and body content.
Upsert a document into Qdrant vector database collection.
Create or update a lead record in CRM with email, company, intent, and other metadata.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear risk classifications (READ_ONLY vs WRITE). LLMs cannot infer retry safety or side effects without explicit hints.
Generic tool descriptions lack LLM-optimization. E.g., 'Create a new Notion page with title, content, and properties' does not explain WHEN to use this vs other tools, what properties are required, or what the response contains. Baseline: 50-200 chars, actionable.