Loading ToolBench
An open-source AI workspace with persistent memory, powered by Novyx. A markdown note-taking app with knowledge graph, AI memory integration, and local-first/self-hosted capabilities.
This server has severe definition quality issues. 21 tools are defined but most lack proper schemas and descriptions. Tool names are inconsistent (mix of verbs like 'rewrite', 'expand', 'summarize' alongside HTTP routes like 'POST /api/auth/callback'). Descriptions are present but often generic (e.g., 'Rewrite selected text' is 24 chars, below the 50-200 char baseline for LLM optimization). Input schemas exist for some tools but are minimal, most parameters lack type constraints, enums, or validation guidance. No output schemas are documented. Error handling is absent from descriptions. Security concerns: tools like 'PUT /api/auth/novyx-key' and 'POST /api/auth/provision' expose authentication flows as tools without permission gates or audit trail guidance. The server conflates UI actions (SelectionToolbar tools) with API endpoints, creating a confused tool taxonomy.
Get masked Novyx API key for current user
Get Stripe Customer Portal URL for managing subscription
Get morning briefing with context, insights, memory drift, and pending tasks
List control actions from Novyx with optional filtering
Get explanation for a specific control action
OAuth callback handler for Supabase authentication
Provision a new Novyx API key for the current user
Tool naming is inconsistent and violates verb_noun convention. UI actions use lowercase verbs (rewrite, expand) while API endpoints use HTTP method + path (POST /api/auth/callback). LLMs cannot parse HTTP routes as tool names, they expect verb_noun format like 'create_auth_session' or 'get_api_key'. This forces the LLM to reason about HTTP semantics instead of intent.
Descriptions are too short and lack LLM-optimization guidance. 'Rewrite selected text' (24 chars), 'Load all notes from storage' (27 chars), 'Sign out the current user' (25 chars) are below the 50-200 char baseline. They do not explain WHEN to use the tool, what it returns, or dependencies. LLMs cannot distinguish between similar tools (e.g., loadNotes vs loadNote) without richer context.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 39 | 2026-07-28+ | v2 |
Sign out the current user
Create a Stripe checkout session via Novyx Core
Stream chat completions with note context, memory recall, and related notes
Test AI provider connection and model availability
Submit a control action to Novyx
Update Novyx API key for current user
Expand selected text
Fix selected text (spelling/grammar)
Flush pending note saves to storage
Load a specific note by path
Load all notes from storage
Rewrite selected text
Save a note with debounced persistence
Summarize selected text
No output schemas are documented for any tool. LLMs cannot plan downstream calls or extract required fields (e.g., does loadNotes return note_id, path, content, or all three?). This forces agents to guess at response structure and risks failed chaining.
Authentication and credential tools (POST /api/auth/provision, PUT /api/auth/novyx-key) lack permission gates and audit trail guidance. No description states who can call these tools, what permissions are required, or how calls are logged. This violates the secret-injection and permission-gate patterns.
Parameter descriptions are minimal or absent. 'selectedText' (The text to rewrite) lacks guidance on length limits, encoding, or format. 'provider' in POST /api/chat is described as 'AI provider config' but does not specify required fields (baseURL, apiKey, model) or constraints. LLMs cannot validate inputs without explicit constraints.