Loading ToolBench
Hardened MCP server exposing SUM verifiable knowledge distillation tools: extract, attest, verify, inspect, render, schema, and meaning-layer receipt family (verify_receipt, meaning_diff, depth_frontier, mint_meaning_receipt, mint_chain_receipt). Supports legacy inline surface plus bind-aware agent surface.
SUM MCP server has 17 tools with mostly complete schemas and descriptions. Naming is domain-specific and verb-forward (extract, attest, verify, inspect, render, mint_*). Descriptions are detailed and technical (50-250 chars per tool, meeting baseline averages). However, several issues reduce the score: (1) Parameter descriptions occasionally lack format/constraint details (e.g., 'private_jwk' in mint_meaning_receipt says 'Private JWK in JSON form (OKP, Ed25519, with 'd' field)' but does not clearly state that the field is REQUIRED in every variant or the exact schema format expected). (2) Output schemas are documented in descriptions but not formally specified as JSON Schema objects in the tool definitions, LLMs cannot parse these descriptions to infer return types. (3) Error handling is mentioned in descriptions ('returns error_class="schema"') but there is no structured error recovery guidance, LLMs cannot know whether to retry, ask the user, or abort. (4) Several 'bind' variants (extract_bind, attest_bind, verify_bind, render_bind, inspect_bind) are separated tools rather than optional parameters on base tools, violating the single-responsibility composition pattern. (5) No rate-limiting, pagination, or result-capping guidance despite tools accepting unbounded text input (MAX_TEXT_CHARS is hard-capped but not enforced at the schema level). Tools are well-named and descriptions provide adequate context, but lack the LLM-optimized structure (explicit return schemas, error classification, idempotency hints) expected at 70+.
Return the bind-aware agent surface manifest: endpoint URLs, hashing algorithm, and tool roster for swarm coordination.
Extract triples and produce a signed CanonicalBundle. Mirrors `sum attest`. Bundle bytes are byte-identical to the CLI path, so they verify through every existing Python/Node/browser SUM verifier.
Content-addressed variant of attest: returns (bundle_hash, axioms) so callers reference bundles by hash instead of embedding full payloads.
Render the whole faithful-to-compressed ladder with per-rung diffs (like `sum depth-diff`). Returns the complete depth frontier with meaning measurements at each rung.
Extract (subject, predicate, object) triples from prose.
Content-addressed variant of extract: returns (triples, content_hash) so callers avoid round-tripping full payloads in agent swarms.
Output schemas not formally specified as JSON Schema. Tool descriptions document return types in prose ('returns a boolean verdict plus caveat and metadata', 'returns the reconstruction plus per-word distillation scores'), but LLMs cannot parse prose to infer return structure. Agents cannot plan downstream tool calls without knowing field names.
Duplicate tool families (extract/extract_bind, attest/attest_bind, verify/verify_bind, render/render_bind, inspect/inspect_bind) violate single-responsibility principle. Should use optional parameters on base tools (e.g., attest(text, extractor, branch, title, signing_key, content_addressed=false) returning either full bundle or hash). Separating them forces agents to reason about which variant to call, increasing error likelihood.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 61 | 2026-07-28+ | v2 |
Inspect a CanonicalBundle: return its axioms, metadata, and (if signed) the signer identity and signature bytes.
Content-addressed variant of inspect: accepts bundle by hash reference.
Per-document kept / dropped / added readout under a named entailment judge (like `sum meaning-diff`). Returns a MEASUREMENT under the named judge, not a certified bound. Proxy is blind to arrangement, sound, connotation, implicature. Judge calls serialise behind one in-process lock; for true parallel judging run N server processes.
Compose hop receipts into a certified chain. Caller supplies private JWK ONLY. Verifies each hop's signature and chains them cryptographically.
Guided issuance of a meaning-risk receipt. Caller supplies private JWK (Ed25519 OKP) ONLY; server never generates, stores, or logs key material. A request without a usable private key is refused with error_class="schema".
Render a CanonicalBundle into a human-readable tome (justified prose). Returns the reconstruction plus per-word distillation scores and (if the Worker is configured) a render_receipt.
Content-addressed variant of render: accepts bundle by hash reference.
Return the JSON schema for CanonicalBundle.
Verify a CanonicalBundle's cryptographic signature and meaning-loss bound. Returns a boolean verdict (verified=true|false) plus caveat and metadata.
Content-addressed variant of verify: accepts bundle by hash reference instead of full payload.
Verify a sum_verify schema receipt (meaning-risk, render, transform, or chain). Dispatches exactly like `python -m sum_verify` (Stage A + optional side-band replay); returns the same honest verdict shape the CLI prints, proxy caveat included. Pure crypto + integer math; safe under arbitrary parallelism.
Parameter 'private_jwk' in mint_meaning_receipt and mint_chain_receipt lacks detailed format specification. Description says 'Private JWK in JSON form (OKP, Ed25519, with 'd' field)' but does not state: (1) exact JSON structure (should 'd' be a base64url-encoded scalar or a string?), (2) whether the 'd' field is always present or optional in variants, (3) whether 'kty', 'crv', 'x' are required or inferred. This forces LLMs to guess at valid formats.
No error classification or recovery guidance. Tools mention error returns ('error_class="schema"' for empty input to extract, 'refused with error_class="schema"' for missing private_jwk), but descriptions do not categorize errors as retryable (e.g., network timeout), user-fixable (e.g., malformed JWK), or fatal (e.g., invalid cryptographic parameters). Agents cannot decide whether to retry, prompt the user, or abort.
MAX_TEXT_CHARS and MAX_LOSSES, MAX_HOPS, MAX_JWKS_KEYS limits mentioned in descriptions but not enforced as JSON Schema constraints (maxLength, maxItems, etc.). LLMs cannot read prose limits reliably, they may pass oversized inputs expecting graceful truncation rather than rejection.
Tools accepting 'extractor' parameter ('auto', 'sieve', or 'llm') do not clearly state preconditions or side effects. E.g., 'llm' is 'disabled unless SUM_MCP_ALLOW_NETWORK=1', but descriptions do not advise what error to expect if the environment variable is unset, or how to guide the user to enable it. This is a user-fixable error; agents need recovery hints.
No idempotency hints via tool annotations (idempotentHint). Tools like verify, inspect, verify_receipt, meaning_diff, depth_frontier are read-only and idempotent, but lack formal markers. Mint_* tools are stateful (they sign and produce new cryptographic objects); without annotations, agents cannot safely retry on ambiguous failures.
No rate-limiting or result-capping guidance despite tools accepting potentially large inputs and producing detailed outputs. E.g., render returns 'per-word distillation scores', for a 200k-char input, this could be 50k scores, exhausting context windows. Descriptions should set agent expectations ('typical output is 1-5 pages')
Some tool descriptions use domain jargon ('faithful-to-compressed ladder', 'axioms', 'entailment judge') without explaining when or why to use the tool from an agent's perspective. E.g., 'depth_frontier' says 'Render the whole faithful-to-compressed ladder with per-rung diffs', but does not answer: When should an agent call this instead of render? What is the use case?