Loading ToolBench
Connect JEV to MCP clients and compare its judgments against general-purpose LLMs using shared datasets and measurable accuracy.
Single tool 'evaluate' has a well-structured schema with comprehensive parameter descriptions and type constraints. Tool name is clear and action-oriented. Description is detailed (194 chars) and explains the three judgment types. Input schema uses Zod with proper union types and validation. Criteria validation is sophisticated, checking type-specific rules (noul/choice/score). Tool annotations (readOnlyHint, idempotentHint, openWorldHint) are present. However, output schema is not explicitly documented, the tool returns raw API bytes as text, which is opaque to LLMs. Error handling is good (explains API errors with recovery guidance) but lacks structured error categorization. No pagination or result limits documented, though the tool itself is stateless.
Judge some content against one or more typed questions and get back probabilities rather than prose — a yes/no likelihood (noul), a pick from a named set (choice), or a position on ordered levels (score). Use it wherever you would otherwise ask a model for an answer and then parse the reply.
Output schema not documented. Tool returns raw JSON bytes as text; LLMs cannot parse structure without explicit schema documentation.
No pagination or result limits documented. While the tool itself is stateless, the API response structure and any limits should be explicit.
Error handling lacks structured categorization. Errors are explained but not classified as retryable/user-fixable/fatal, limiting agent recovery planning.
API key (TYPESAFE_API_KEY) is injected server-side, which is correct, but no explicit documentation of permission scopes or what the tool can/cannot do.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 64 | 2026-07-28+ | v2 |