Loading ToolBench
The soit MCP server exposes 4 tools with HTTP transport. Tools have descriptions and mostly documented input parameters, but quality is inconsistent. knowledge_query is complex with 10 parameters, some underdescribed and redundantly overlapping (ctx object vs individual tenant_id/workspace_id/user_id). Output schemas are not documented for any tool. Error handling and recovery guidance are absent. The builtin tools (time_now, random_int) are minimal but well-named. create_review_ticket is write-capable but lacks confirmation or dry-run pattern. Descriptions average ~90 characters, below the 194-char production baseline. Parameter types are present but several lack contextual descriptions (e.g., 'strategy' on knowledge_query has no guidance on valid values). Overall, definitions are functional but fall short of production-grade quality expected for agent-facing tools.
Create a deterministic demo review ticket with a generated ticket ID.
Build a scoped runtime service and return knowledge retrieval results.
Return a random integer within range.
Return current UTC time.
Output schemas not documented for any tool. LLMs cannot infer what fields to expect, forcing them to guess and risking context loss or failed downstream tool chaining.
knowledge_query has 10 parameters with overlapping semantics. Both 'ctx' object (containing tenant_id, workspace_id, user_id) and individual tenant_id, workspace_id, user_id parameters are present. Parameter relationships are undocumented (rubric C.5), forcing LLMs to guess which to use. Redundancy invites misuse.
Several parameters lack descriptive context. knowledge_query's 'strategy' parameter has no guidance on valid values; 'filter' is underspecified; random_int's min/max lack range constraints (e.g., maximum distance allowed).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 63 | 2026-07-28+ | v2 |
create_review_ticket is a write operation (risk=WRITE) but lacks confirmation step, dry-run pattern, or explicit error guidance. Agents can accidentally create tickets without review.
No error handling or recovery guidance across any tool. Error responses do not tell LLMs what to do next (e.g., 'User not found. Try search_users() with a partial name.'); no error categorization (retryable vs. fatal).
Tool descriptions are below production baseline (avg 90 chars vs. baseline 194 chars). Many lack context on WHEN to use the tool or its prerequisites.
create_review_ticket's 'payload' parameter overlaps with explicit params (customer_id, priority, message, url). Unclear whether payload should override or supplement explicit params. Undocumented parameter relationships (rubric C.5).