Loading ToolBench
MCP server for TypeSafe AI's Jev model, offering tools for evaluating text against questions, validating requests, and batch processing
jev-cli exposes 7 tools for probabilistic evaluation via the Jev API. Tool naming is clear and verb-driven (evaluate, noul, choice, score, validate, list_models, batch_run). Descriptions are comprehensive and detailed, often 200+ characters, explaining WHAT each tool does, WHEN to use it, and key constraints (e.g., 'Jev never generates text: it returns calibrated probabilities'). However, input schemas are largely absent from the visible source code. Only list_models shows an explicit schema (empty object). The other 6 tools have descriptions but no visible parameter definitions, type constraints, or enums in the provided code excerpt. This forces a schema score of 0 for most tools. Error handling is implicit (validation errors listed in tool descriptions) but not formalized in response structures. Composition is strong: tools are single-purpose and chainable (validate before evaluate). Security is good: no credentials exposed as parameters.
One question set over every row of a file.
Pick the one option that best fits a `state`, from up to 255 options you define in `criteria` (option name to description, or null when the name says enough). Returns `choice` (the winner), `confidence` (how peaked the distribution is, not how correct it is) and `probabilities` per option. Choose choice for a relative judgment, "which one of these?". Always include an escape option such as `other`, `none_of_the_above` or `not_stated`: Jev cannot pick an option you left out. Use `noul` per label when several can apply at once, `score` for an ordered scale, and `evaluate` for several questions about the same state. The result is the same JSON envelope `jev` prints: `model` (the versioned model that answered), `requested_model`, `usage`, `cost_usd` (estimated; null when the price is unknown), `request_id`, `latency_ms`, `warnings` (validation lints, only when there are any) and `session` (calls made and estimated spend so far in this session). The request is validated offline first: an invalid one is a tool error listing every finding, and nothing is sent. Jev never generates text: it returns calibrated probabilities. It cannot count, do arithmetic or compare numbers, and it reads dates as text (ask for the parts, such as the month, as choices, and compare in code). It is a literal reader: put the full meaning in `instructions`, since question ids are never shown to it. Ask one snap judgment per question; split anything that needs reasoning into several questions. In a choice, always include an escape option such as `other` or `not_stated`.
Evaluate one `state` against several typed questions in a single call: `noul` (yes/no), `choice` (one of a set of options) and `score` (a rubric of 2 to 10 levels), keyed by ids you choose. Prefer this over `noul`, `choice` and `score` whenever there is more than one question about the same state, including speculative ones: questions are answered in parallel, and one call is far cheaper and faster than several. The result is the same JSON envelope `jev` prints: `model` (the versioned model that answered), `requested_model`, `usage`, `cost_usd` (estimated; null when the price is unknown), `request_id`, `latency_ms`, `warnings` (validation lints, only when there are any) and `session` (calls made and estimated spend so far in this session). The request is validated offline first: an invalid one is a tool error listing every finding, and nothing is sent. `answers` holds one answer per question id, exactly as the API returned it. Jev never generates text: it returns calibrated probabilities. It cannot count, do arithmetic or compare numbers, and it reads dates as text (ask for the parts, such as the month, as choices, and compare in code). It is a literal reader: put the full meaning in `instructions`, since question ids are never shown to it. Ask one snap judgment per question; split anything that needs reasoning into several questions. In a choice, always include an escape option such as `other` or `not_stated`.
Input schemas missing for 6 of 7 tools. Only list_models has a visible schema definition. evaluate, noul, choice, score, validate, and batch_run lack explicit parameter type definitions, enums, and constraints in the source code.
batch_run has a minimal description ('One question set over every row of a file.') that lacks context on WHEN to use it, what file formats are accepted, and what the output structure is. Descriptions under 20 chars cannot exceed 40 points.
No documented output schemas for any tool. LLMs cannot plan downstream calls or extract fields without knowing the response structure (e.g., what fields does evaluate return? Is it paginated?).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 43 | 2026-07-28+ | v2 |
List the model names and aliases the account can use, with a description and release date: `{"models": [{"name", "description", "release_date"}]}`. Use it to pick a `model` for the other tools. Aliases such as `jev-latest` move to newer versions without notice; a versioned id such as `jev-1.13.0` is accepted even when it is not listed, and every evaluation reports the versioned `model` that answered.
Ask one yes/no question about a `state`. Returns `noul`, the probability that the answer is yes (0 to 1). Choose noul for an absolute judgment, "does this hold?", which may be low for every candidate; for tagging, ask one noul per label (in one `evaluate` call). A value near 0.5 means Jev cannot tell, not "medium". Use `choice` when exactly one of several options fits, `score` for a position on an ordered scale, and `evaluate` for several questions about the same state. The result is the same JSON envelope `jev` prints: `model` (the versioned model that answered), `requested_model`, `usage`, `cost_usd` (estimated; null when the price is unknown), `request_id`, `latency_ms`, `warnings` (validation lints, only when there are any) and `session` (calls made and estimated spend so far in this session). The request is validated offline first: an invalid one is a tool error listing every finding, and nothing is sent. Jev never generates text: it returns calibrated probabilities. It cannot count, do arithmetic or compare numbers, and it reads dates as text (ask for the parts, such as the month, as choices, and compare in code). It is a literal reader: put the full meaning in `instructions`, since question ids are never shown to it. Ask one snap judgment per question; split anything that needs reasoning into several questions.
Rate a `state` on an ordered rubric of 2 to 10 levels, lowest first, given in `criteria`. Returns `score` (the expected level, from 0 to the number of levels minus 1), `confidence`, `probabilities` and `legend`, both keyed by the level number as a string. Each level is judged on its own, so describe a situation for each ("answers the question with a working example"), not a degree ("better than the previous"); rate one dimension per score, and do not use it to extract a number. Use `choice` for unordered options, `noul` for yes/no, and `evaluate` for several questions about the same state. The result is the same JSON envelope `jev` prints: `model` (the versioned model that answered), `requested_model`, `usage`, `cost_usd` (estimated; null when the price is unknown), `request_id`, `latency_ms`, `warnings` (validation lints, only when there are any) and `session` (calls made and estimated spend so far in this session). The request is validated offline first: an invalid one is a tool error listing every finding, and nothing is sent. Jev never generates text: it returns calibrated probabilities. It cannot count, do arithmetic or compare numbers, and it reads dates as text (ask for the parts, such as the month, as choices, and compare in code). It is a literal reader: put the full meaning in `instructions`, since question ids are never shown to it. Ask one snap judgment per question; split anything that needs reasoning into several questions.
Check a request offline, with every rule `evaluate` applies plus lints for common mistakes (a choice without an escape option, a misspelt field). Nothing is sent, nothing is billed, and no API key is needed. Takes the same arguments as `evaluate`, with `state` optional (give it to get a size estimate). Returns `valid` (false only when there are errors; warnings do not count), `summary` with the counts, `findings` (each with `severity`, `rule`, `question`, a JSON Pointer `path`, `message` and `suggestion`) and `size`, the estimated input tokens. Use it while writing or fixing a request; `evaluate`, `noul`, `choice` and `score` run the same checks before every call anyway.
Error handling is described in tool descriptions but not formalized. No explicit error classification (retryable vs. user-fixable vs. fatal) or recovery guidance in response structures.
Tool descriptions embed example values and constraints (e.g., 'up to 255 options', 'Jev never generates text') in prose rather than as formal schema constraints. LLMs cannot reliably parse these from text.