Loading ToolBench
Server has 9 tools with basic schemas and descriptions, but significant gaps limit production readiness. All tools have descriptions (10-100 chars, mostly adequate), and input schemas are present with type definitions. However, output schemas are undocumented, LLMs cannot predict return structures. Session ID management via manual dict is a critical anti-pattern: credentials leak into logs and traces. Parameter descriptions lack constraints (e.g., no format specs for host URLs, no enum for project fields). Error handling is minimal, most exceptions re-raise without recovery guidance. No tool annotations (readOnlyHint, destructiveHint). Naming is clear (verb_noun pattern), but composition issues exist: list_projects and list_all_projects are nearly identical, forcing LLM disambiguation. Missing pagination parameters despite returning lists. No idempotency guarantees documented.
Creates a new project.
Creates a new user story in a project.
Gets detailed information about a specific project by its ID.
Gets detailed information about a specific project by its slug.
Lists all projects visible to the user (requires admin privileges for full list). Uses the provided session_id.
Lists projects accessible to the user associated with the provided session_id.
Session ID stored in server-side dict (active_sessions) exposes credentials in logs and traces. Session IDs are secrets and must never be logged or returned to clients. Use stateless token validation or server-side session stores with encryption.
Output schemas undocumented. list_projects, list_all_projects, list_user_stories return List[Dict[str, Any]] with no field documentation. LLMs cannot predict structure, forcing trial-and-error field access. Document return types with field names, types, and descriptions.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 56 | <=2025-11-25 | v2 |
Lists user stories for a specific project.
Logs into a Taiga instance using username/password and returns a session_id for subsequent authenticated calls.
Updates details of an existing project.
list_projects and list_all_projects are nearly identical (both call same underlying method). LLM cannot distinguish when to use each. Either merge into one tool with a scope parameter (enum: 'accessible'|'all') or clearly document the permission difference in descriptions.
No pagination support. list_projects, list_all_projects, list_user_stories return unbounded lists. If a project has 1000 user stories, the response bloats context. Add limit and offset/cursor parameters; document max result count in descriptions.
Parameter descriptions lack constraints. 'host' accepts any string, no format validation (must be HTTPS URL). 'project_id' is integer with no range. 'description' fields unbounded. Add format specs, min/max, and enums where applicable to guide LLM input and enable early validation.