Loading ToolBench
Modular, composable Python client and decision toolkit for the TypeSafe Jev (System One) API, exposed as MCP tools over the official SDK
Server exposes 7 tools with complete input schemas and descriptions. Naming follows verb_noun convention (jev_ask, jev_evaluate, jev_models, etc.). Descriptions are detailed (100-300 chars) and explain WHAT, WHEN, and RETURN. However, parameter descriptions lack actionable constraints (no enums, ranges, or format specs). Output schemas are documented in descriptions but not formally declared. Error handling is minimal, no recovery guidance or categorization. All tools are READ_ONLY (no destructive operations), reducing risk. Composition is sound: tools chain naturally (models → ask → evaluate). No secrets exposed as parameters.
Ask named questions about one state in a single API call. ``questions`` maps question id to either a CLI ask-spec string (``noul:<instructions>``, ``choice:<instructions>:k=desc,...``, ``score:<instructions>:l1,l2,...``) or a native ``{type, instructions, criteria}`` dict. Returns ``{model, answers: {id: answer dict with type}, usage, request_id}``; ``request_id`` is omitted when the API did not return one. ``provider`` selects the registered API provider (default ``jev``).
Expected value of a score distribution over its level indices. ``probabilities`` maps level index (as a string) to probability. With ``weights`` (one per level, positional order matching the sorted level indices) the distribution is re-weighted as in :func:`daf_jev.compose.composite_score`. Requires at least 2 levels; probabilities must be finite and non-negative and keys integer level indices (ValueError otherwise). ``provider`` is validated against the registry; local math is provider-agnostic.
Binary gate on answer confidence: return the answer's choice/noul/score if confidence >= threshold, else return the ``below`` string. ``answer`` is a dict with keys ``type`` (``choice`` | ``noul`` | ``score``), ``choice`` or ``noul`` or ``score``, and ``confidence``. ``provider`` is validated against the registry; local logic is provider-agnostic.
Verify the docs/reference snapshot against its manifest. Returns ``{ok: bool, manifest_path, snapshot_path, n_files, n_missing, n_extra, missing, extra}``. When ``manifest`` is omitted, the default manifest is loaded from :const:`daf_jev.docs_verify.DEFAULT_MANIFEST`. ``provider`` is validated against the registry; verification is provider-agnostic.
Parameter constraints missing: 'probabilities' object lacks format spec (must be finite, non-negative, integer keys); 'threshold' and 'high'/'low' lack numeric ranges (0.0 - 1.0 implied but not stated); 'concurrency' lacks min/max bounds. LLMs cannot infer these from descriptions alone.
Output schemas documented in descriptions but not formally declared in tool registration. LLMs cannot parse unstructured text descriptions to extract field types and structure. Formal JSON Schema output documentation is missing.
Error handling lacks recovery guidance. ValueError exceptions (e.g., 'empty states list', 'no API key', 'unknown provider') are raised but descriptions do not tell LLM what to do next. No categorization of errors as retryable vs. user-fixable.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 66 | 2026-07-28+ | v2 |
Evaluate a fixed question set over many states, concurrently. Returns the aggregate summary only (no per-state records): ``{n_states, n_errors, total_input_tokens, total_output_tokens, mean_latency_s, p95_latency_s, questions}``. Raises ``ValueError`` on an empty ``states`` list or a state that is neither a string nor a JSON object/array. ``provider`` selects the registered API provider (default ``jev``).
List available model cards as dicts. When ``pick`` (``latest`` | ``first`` | ``last``) or ``contains`` (case-insensitive substring filter on the model name; the empty string is no filter) is given, one card is selected via :func:`daf_jev.models.pick_model` and returned as a single-element list. Raises ``ValueError`` when nothing matches. ``provider`` selects the registered API provider (default ``jev``).
Three-way gate on answer confidence: return ``above`` (high confidence), ``below`` (low confidence), or ``review`` (medium confidence). ``answer`` is a dict with keys ``type`` (``choice`` | ``noul`` | ``score``), ``choice`` or ``noul`` or ``score``, and ``confidence``. ``provider`` is validated against the registry; local logic is provider-agnostic.
'questions' parameter in jev_ask and jev_evaluate accepts both CLI strings and native dicts but lacks enum or pattern constraint. LLM cannot validate question specs (noul:, choice:, score: prefixes) without explicit grammar documentation.
Tool descriptions reference external modules and functions (daf_jev.cli.parse_question_spec, daf_jev.models.pick_model) without explaining the behavior in plain language. LLMs cannot follow code references; descriptions must be self-contained.