Loading ToolBench
MCP server surface for the Calma verification engine — re-run a repo, recompute every number it reports, and prove or break the claim.
Calma MCP server has 11 tools with clear, domain-specific names (add_argv, resolve_data_path, etc.) that follow verb_noun conventions. However, critical gaps undermine quality: input schemas are present but lack formal JSON Schema type declarations (parameters are described as Python type hints like 'list[str]' and 'dict' rather than JSON Schema objects with 'type', 'items', 'properties'). Descriptions are present (avg ~120 chars) but many are terse and lack actionable guidance for LLM selection. No output schemas are documented. Error handling is absent, tools provide no recovery guidance. Security concerns: resolve_data_path is marked WRITE but no permission gates or audit trails are visible. Most tools are read-only file/repo operations, which is appropriate, but the MCP server lacks composition patterns (no pagination, no batch operations, no chaining IDs in responses). This is a functional but immature implementation.
Append validated args to an entrypoint argv (A2); path-like args must resolve in the repo.
Top-level import roots of one Python file (wraps build._imported_roots).
Era-pin inferred deps, then relax any pin incompatible with the repo's interpreter (C1).
Ranked entrypoint candidates (EntryCand[]) — wraps build.detect_entrypoint as the top pick and enumerates the rest (A4).
Regex-search repo text files; returns [(rel, lineno, line)] (bounded).
Build the pip spec list + strict flag from a repo's declared manifest (B1).
List repo files (relative paths), skipping vendored/cache dirs.
Input schemas use Python type hints (list[str], dict, callable) instead of JSON Schema format. No 'type', 'items', 'properties', or 'required' fields visible. LLMs cannot parse these to understand parameter structure.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields (e.g., what does find_entrypoints return? A list of strings? Objects with name/path/rank fields?).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 52 | 2026-07-28+ | v2 |
Read a repo-relative text file (path-traversal-guarded, size-bounded).
Prediction files NEWLY created/changed by a run — recapture fresh outputs, not just committed ones (C2).
Make a missing-data run runnable: relink a file already committed under a different path to where the code reads it (free), else fetch a known public dataset. Supplies INPUTS only — never edits compute, so it can't fabricate a verdict.
Merge validated string env vars into an approach's env (A2).
resolve_data_path marked WRITE but no permission gates, audit trails, or confirmation steps visible. Destructive operations should support dry-run or require explicit confirmation.
No error handling guidance. Tools provide no recovery hints (e.g., 'File not found. Try list_tree() to see available files.'). LLMs cannot self-correct on failures.
Descriptions are terse (avg 60 chars for many tools). Missing context on WHEN to use each tool vs. similar ones (e.g., grep vs. list_tree; read_file vs. list_tree). LLMs struggle to select the right tool.