Loading ToolBench
RAG-powered chat API with knowledge bases, file attachments, and vector embeddings using PostgreSQL and OpenAI
Neural Foundry exposes 25 tools with consistent naming (verb_noun pattern) and basic descriptions. However, critical gaps undermine production readiness: parameter descriptions are minimal or absent across most tools, input schemas lack type constraints (enums, ranges), output schemas are undocumented, and error handling provides no recovery guidance. Tool names are clear (login_user, create_chat, delete_kb) but descriptions average ~80 chars, below the 194-char baseline for A+ tools. Parameters like 'user_id', 'chat_id', 'kb_id' lack format/validation hints. No tool documents what it returns or how to chain results. Security concerns: no evidence of permission gates on destructive operations (delete_chat, delete_kb, delete_document). Composition is reasonable (single responsibility per tool) but lacks idempotency guarantees and batch variants.
Attach a knowledge base to a chat session. A chat can have 0, 1, or many KBs attached.
Attach multiple knowledge bases to a chat session at once. Skips KBs that are already attached.
Handle user messages and generate an assistant reply. Now supports Knowledge Base context automatically and returns metadata! Returns reply text and source information (KB, chat history usage).
Create a chat session not linked to any specific user.
Create a new chat session linked to a specific user.
Create a new knowledge base for a given user.
Output schemas completely undocumented. No tool documents what fields it returns, forcing LLMs to guess structure and breaking downstream tool chaining. E.g., conversation_chat returns 'reply text and source information' but no schema for 'source information' structure.
Parameter descriptions missing or trivial across 20+ tools. E.g., 'user_id' described only as 'User ID' with no format hint (UUID? numeric? string?). LLMs cannot validate or construct valid IDs without format guidance.
No input validation constraints. Parameters accept free-form strings with no enums, ranges, or patterns. E.g., 'title' and 'description' fields have no length limits; 'system_prompt' has no guidance on format or max tokens.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 50 | <=2025-11-25 | v2 |
Manually create a new user (admin endpoint). Optional - since login now auto-creates users.
Delete a chat session and its messages.
Delete a file attachment from a chat. This removes all associated chunks (CASCADE).
Delete a document from knowledge base. This will also delete all associated chunks (CASCADE).
Delete a knowledge base and all its documents/chunks.
Detach a knowledge base from a chat session.
Fetch all chat sessions with their messages.
Fetch all messages for a given chat_id, properly ordered by timestamp.
Get user details by ID.
Fetch all chats belonging to a specific user.
List all file attachments in a chat.
List all knowledge bases attached to a specific chat session. Returns empty list if no KBs are attached.
List all chat sessions that are using a specific knowledge base. Useful to see which chats depend on a KB before deletion.
List all documents in a knowledge base.
List all knowledge bases created by a specific user with document count.
List all users in the system.
Login with username. Auto-creates user if username doesn't exist. This enables self-registration on first login.
Upload a temporary file attachment to a chat. These files are processed and used for context in this chat only.
Upload a file to the KB. Embedding ingestion runs asynchronously.
Destructive operations (delete_chat, delete_kb, delete_document, delete_chat_attachment) lack permission gates and confirmation patterns. No evidence of authorization checks or dry-run support. Agents can delete data without safeguards.
No error recovery guidance. Tool descriptions do not explain failure modes, retryability, or next steps. E.g., 'upload_file_to_kb' runs 'asynchronously' but does not document how to check status or handle timeouts.