Loading ToolBench
Language-aware FTS preprocessing across 22 supported language codes, with zero required dependencies.
babel-memory is a specialized language-detection and tokenization library with 12 well-named, read-only tools. Naming is strong (verb_noun pattern: detectLanguage, tokenizeForFts, initTokenizer). Descriptions are present and moderately detailed (avg ~150 chars), explaining WHAT each tool does and WHEN to use it. However, parameter descriptions are sparse or missing for several tools (e.g., initTokenizer.opts lacks detail on what languages are available; segmentWithIntl.locale lacks format guidance). Output schemas are not explicitly documented in the source, only input schemas are visible. Error handling is minimal; no recovery guidance or actionable error messages are evident. The library is read-only (no destructive operations), which reduces security risk but also means no confirmation patterns are needed. Overall, this is a solid domain-specific tool set, but lacks the polish of production-grade agent tools: parameter constraints are underspecified, output structures are not formally documented, and error paths are not guided.
Detect the primary language of text based on Unicode script analysis. Returns one of: zh, ja, ko, th, ar, hi, ru, en.
Like detectLanguage, but also reports per-script ratios and whether the text significantly mixes scripts. Useful for AI conversation logs where CN/EN (or JA/EN) mixing is the norm.
Like detectLanguage, but refines Latin-script text into specific language codes (de, fr, es, ...) when tinyld is installed. Non-Latin scripts are resolved by Unicode analysis. Returns ISO 639-1 codes as plain strings.
Get knowledge graph extraction prompts in the specified language. Returns system prompt and user template for extracting (subject, predicate, object, confidence) tuples from text.
Diagnostic: which optional packs are actually loaded right now. Useful for verifying deployment configuration.
Get session summarization prompts in the specified language. Returns system prompt and dimension labels for structured conversation analysis.
Output schemas not documented. Tools return structured data (e.g., detectLanguageDetailed returns per-script ratios, getKgPrompt returns system prompt + user template) but no formal output schema is visible in source. LLMs cannot plan downstream calls or extract fields without knowing return structure.
Parameter descriptions lack format/constraint guidance. initTokenizer.opts.languages says 'Only load tokenizers for these language codes' but does not list valid codes (zh, ja, ko, th, ar, hi, ru, en, or ISO 639-1). segmentWithIntl.locale says 'Locale code' but does not specify format (e.g., 'zh' vs 'zh-CN'). LLMs cannot validate inputs without explicit constraints.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 41 | 2026-07-28+ | v2 |
Initialize available tokenizers in parallel. Non-fatal if any fail -- each will fall back gracefully. Safe to call multiple times (idempotent).
ICU segmentation + bigram expansion for zh/ja. ICU emits whole compounds which breaks BM25 partial matches. Appending bigrams for CJK tokens of length >= 3 restores recall — the same approach lunr-languages uses.
Diagnostic: whether the optional tinyld detector is loaded.
Load the optional tinyld language detector. Called by initTokenizer(); safe to call directly and idempotent. Returns true when available.
Word-segment text using the runtime's built-in ICU dictionaries. Returns null when Intl.Segmenter is unavailable or produces no word-like tokens, so callers can fall through to the next strategy.
Pre-tokenize text for BM25 FTS indexing. Input is NFKC-normalized first, then routed by language: zh uses jieba search-mode segmentation (fallback: ICU+bigrams, then char split); ja uses kuromoji segmentation (fallback: ICU+bigrams, then char split); ko uses character-level split; th uses wordcut segmentation (fallback: ICU word segmentation); Snowball languages use word-level stemming; en/unknown use passthrough with embedded CJK/Hangul/Thai runs segmented. IMPORTANT: apply the same call to both indexed text AND query.
No error handling or recovery guidance. Tools are read-only, so no destructive errors, but no guidance on what to do if tinyld fails to load, if Intl.Segmenter is unavailable, or if a language code is unsupported. segmentWithIntl returns null on failure but callers are not told how to handle it.
Diagnostic tools (isTinyldLoaded, getLoadedTokenizers) lack context on when to call them. Descriptions say 'Diagnostic' but do not explain the user intent: 'Call this to verify deployment configuration' or 'Use this to check if optional dependencies are available before calling tokenizeForFts'.
tokenizeForFts description is verbose (>500 chars) and buries key details. It lists fallback chains (jieba → ICU+bigrams → char split) but does not clearly state: 'Apply the same call to both indexed text AND query' is buried at the end. Descriptions should be 50-200 chars for LLM clarity.