Loading ToolBench
Ayaka has well-structured tool definitions with clear naming (verb_noun pattern) and comprehensive descriptions (100-250 chars, well above the 10-char minimum). All 9 tools have input schemas with typed parameters and descriptions. However, output schemas are not documented, tools return JSON strings without declaring the structure of those strings. Parameter descriptions are good but lack explicit constraints (e.g., no min/max for 'days' or 'limit'). Error handling is minimal; tools do not guide recovery or categorize failures. No tool annotations (readOnlyHint/destructiveHint) despite clear read/write distinction. The server logs tool calls but does not validate inputs early or provide actionable error messages.
Items not seen in the last N days that may need review. Returns up to 20 items per category, sorted oldest first. Call at the start of a session to weave review into conversation naturally.
Look up one or more grammar patterns. Tries exact match first, then partial (LIKE). Returns reference data (meaning, example sentence, JLPT level) and learner progress (status) for each. Patterns are stored as plain Japanese text, e.g.: N5: "がある", "から 1", "いちばん", "する" N4: "後で", "あまり○○ない", "お○○ください" N3: "あまりに", "一方だ", "いくら○○ても / いくら○○でも" N2: "あるいは", "いきなり", "以上に" N1: "あえて", "あくまでも", "案の定" Disambiguated meanings use numbered suffixes: "が 1" (subject), "が 2" (but). Placeholders use ○○ for variable parts.
Items introduced within a date range. Useful for reviewing what was learned recently or during a specific period.
Coverage summary for a JLPT level across kanji, vocab, and grammar. Shows total reference items, tracked items, and breakdown by confidence/status. Use this to gauge overall learner progress before planning lessons.
Look up one or more kanji characters. Returns reference data (meanings, on/kun readings, JLPT level, grade, frequency, stroke count) and learner progress (confidence, times_seen, produced) for each.
Output schemas not documented. Tools return JSON strings (e.g., get_jlpt_coverage returns dict serialized to JSON) but the structure of those strings is not declared. LLMs cannot plan downstream calls or extract fields without reverse-engineering the output.
No tool annotations (readOnlyHint/destructiveHint). The server clearly distinguishes read-only tools (get_*) from write tools (update_*), but does not declare this via tool annotations. LLMs cannot infer safety without explicit hints.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 58 | <=2025-11-25 | v2 |
Look up one or more vocabulary words by written form or reading. Matches against the word column (kanji form) or reading column (kana). Returns reference data (meaning, part of speech, JLPT level) and learner progress (confidence, produced) for each.
Update learner progress for a grammar pattern. Creates a new record on first encounter. Pattern must exactly match an entry in grammar_ref.
Update learner progress for a kanji. Creates a new record on first encounter. Increments times_seen automatically each call.
Update learner progress for a vocabulary word. Creates a new record on first encounter.
Parameter constraints missing. 'days' (get_due_for_review) and 'limit' (get_introduced_between) lack min/max bounds. 'jlpt_level' accepts 1-5 but no validation is documented. LLMs may pass invalid values (e.g., days=999, jlpt_level=10).
No error recovery guidance. Tools do not return actionable error messages. If a pattern lookup fails (get_grammar), the LLM receives no hint about what to try next (e.g., 'Pattern not found. Try a partial match or list available patterns.').
Result limits not enforced or documented. get_due_for_review and get_introduced_between cap results at 20 items per category, but this limit is not stated in the tool description. LLMs may expect more results and retry unnecessarily.