Loading ToolBench
A Flask-based team memory system with SQLite local storage, DynamoDB team sync, ChromaDB semantic search, and savings tracking. Provides FTS and RAG search modes for retrieving shared knowledge.
Server has 14 tools with descriptions and basic schemas, but significant quality gaps. Most tool names lack clear action verbs (e.g., 'ddb_sync push' instead of 'push_to_ddb'). Parameter descriptions are present but often generic. Input schemas visible for most tools but lack proper type constraints and enums. Output schemas are not documented. Error handling is minimal, no recovery guidance or actionable error messages. Tool composition is poor: 'ddb_sync' and 'rag_sync' are command-style tools with subcommands rather than discrete, composable tools. No tool annotations (readOnlyHint, destructiveHint). Security: no evidence of permission gates or audit logging.
Pull shared DynamoDB memories into local SQLite with incremental delta sync. Only fetches rows newer than last pull timestamp. Rebuilds FTS index and re-syncs ChromaDB.
Push local SQLite memories to shared DynamoDB table with embeddings. Validates confidence levels and enforces evidence-backed confidence rules.
Semantic vector search on shared DynamoDB using native SearchVectors API. Searches across known scopes with configurable top-K results.
Show row counts comparison between local SQLite and shared DynamoDB.
Check if a new memory being saved conflicts with or duplicates an existing one using semantic similarity threshold.
Semantic search with weighted scoring combining similarity, confidence, recency, and corroboration. Increments times_surfaced counter for top results.
Auto-generate one summary row per scope from its 5 most-recent non-summary rows. Marks old summaries as stale.
Tool naming violates verb_noun convention. 'ddb_sync push', 'rag_sync sync', 'savings_tracker log' use command-style subcommands instead of clear action verbs. Should be 'push_to_ddb', 'sync_rag_index', 'log_task_savings'. LLMs struggle to parse intent from compound names.
No input schemas visible for 'ddb_sync push', 'ddb_sync pull', 'ddb_sync status', 'rag_sync sync', 'rag_sync summarize', all show empty {} input. These tools accept no parameters, making them inflexible and hard to compose.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 41 | 2026-07-28+ | v2 |
Index all non-stale memories from SQLite into ChromaDB vector store. Computes content hashes to skip unchanged rows. Removes stale vectors.
Log a completed task with estimated time and token savings. Writes to local SQLite and optionally to shared DynamoDB.
Log latency comparison between DDB native and ChromaDB backends for a query.
Log developer onboarding with setup type and hours saved. Writes to local SQLite and optionally to shared DynamoDB.
Print or export savings summary aggregated from team DynamoDB or local SQLite only.
Search memories using FTS (full-text search), RAG (semantic similarity), or recent mode. Supports smart mode that tries FTS first then falls back to RAG.
Auto-generate summary bullets from memory content including ticket references, type labels, key sentences, status, and technical terms involved.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields. E.g., 'search' returns results array but structure is not formally specified.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). 'ddb_sync push', 'ddb_sync pull', 'rag_sync sync' are destructive writes but not marked. Agents cannot distinguish safe reads from risky mutations.
Error handling is absent. No recovery guidance, no actionable error messages, no error classification (retryable vs fatal). E.g., 'rag_search' returns ([], str(e)) but LLM has no guidance on what to do next.