Local semantic search over personal knowledge bases
Two tools with basic definitions and schemas. semantic_search has a clear description (127 chars) and three typed parameters with descriptions. list_sources has a minimal description (67 chars) and no parameters. Both tools return unstructured strings rather than structured objects, forcing LLMs to parse free-text output. No error handling guidance, no output schema documentation, and no parameter constraints (e.g., limit bounds). Naming is verb-forward (semantic_search, list_sources) but descriptions lack WHEN/WHY context and recovery guidance.
List all available knowledge base sources that have been indexed.
Search the user's personal knowledge base. Use this BEFORE answering questions about security testing, vulnerabilities, or techniques, to supplement your knowledge with the user's notes and documentation.
Output is unstructured string, not structured JSON. LLMs must parse free-text markdown to extract result metadata (source_id, similarity score, chunk index). This wastes tokens and invites parsing errors.
semantic_search parameter 'limit' has no bounds documented. Description says 'Maximum number of results' but does not specify min/max (e.g., 1 - 100). LLMs may pass absurd values.
No error handling guidance. If search returns no results, tool returns 'No results found.' but does not suggest next steps (e.g., 'Try a broader query' or 'Check list_sources() to verify indexed data'). Agents cannot recover.
semantic_search description lacks WHEN/WHY context. It says 'Search the user's personal knowledge base' but does not explain when to call it vs. other tools, or what prerequisites exist (e.g., 'Requires sources to be synced via 'lake sync' first').
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 46 | 2026-07-28+ | v2 |
list_sources description is 67 chars, below the 100-char baseline for discovery tools. Does not explain what structure is revealed or when to call it (e.g., 'Call this first to see available sources before searching').