Loading ToolBench
Single tool 'chat' with a comprehensive schema and extensive description. The tool has a well-formed JSON Schema with proper type definitions and enums. Description is detailed (1200+ chars) and explains the workflow, context management, and file attachment best practices. However, some parameter descriptions could be more concise and actionable for LLM parsing. Schema includes proper constraints (enum for model, reasoning_effort) and required fields. The tool is idempotent for read operations but stateful (manages continuation_id), which is appropriate. Output schema is not explicitly documented in the tool definition itself, this is a notable gap for an LLM to understand what fields to expect in the response.
Direct access to state-of-the-art AI models via OpenRouter. Provide EXACTLY ONE: - title: Start fresh (when switching topics, context too long, or isolating model contexts) - continuation_id: Continue existing conversation (preserves full context) When starting fresh: Model has no context - include background details or attach files When continuing: Model has conversation history - don't repeat context Workflow: 1. First call: Use 'title' → receive continuation_id in response 2. Follow-ups: Use the continuation_id from previous response 3. New conversation thread: Start fresh with a new title FILE ATTACHMENT BEST PRACTICES: - Proactively attach relevant files when starting new conversations for context - For long content (git diffs, logs, terminal output), save to a file and attach it rather than pasting verbatim in prompt - Files are processed more efficiently and precisely than inline text CRITICAL - PROVIDE FULL CONTEXT: When starting a new conversation with any model, you MUST provide complete context. The model has no knowledge of: - Previous discussions with other models - Code files you've been working with - The problem background or constraints WRONG approach (context loss): → GPT: "Gemini suggested Option A. What do you think?" CORRECT approach (full context): → GPT: [Attach relevant code files] "I'm implementing X with constraints Y and Z. I'm considering Option A (because...) vs Option B (because...). Gemini favored Option A citing reason R. What's your analysis?" Each model conversation is isolated. Never assume shared context - always provide: 1. The complete problem statement 2. Relevant code files (use 'files' parameter) 3. Key constraints and requirements 4. Prior conclusions WITH their reasoning (not just "X suggested Y")
Output schema not documented. The tool description does not specify what fields the response contains (e.g., continuation_id, model_response, tokens_used, etc.). LLMs need explicit documentation of response structure to plan downstream operations and extract data.
Tool description is overly verbose (1200+ chars). While comprehensive, it exceeds the baseline of 194 chars and includes redundant formatting instructions that would dilute token efficiency. The core 'What does it do? When? Prerequisites?' framework gets buried.
Parameter 'model' is an enum of short model names, but the description returns full descriptions. Mismatch between enum values (which the LLM will pass) and the description text could cause confusion if the full model path is expected downstream.
Mutual exclusivity of 'title' and 'continuation_id' is stated in descriptions but not enforced in the schema (no oneOf constraint). LLMs may pass both, causing ambiguous behavior.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 52 | <=2025-11-25 | v2 |
| 2026-03-09 | D | 50 | - | v1 |
No error handling guidance in the tool description. What happens if both title and continuation_id are provided? What if the model is unsupported? What if file paths don't exist? LLMs have no recovery path.