Loading ToolBench
Deterministic Python research engine for prediction-market evidence-backed backtests
Batter exposes 16 quantitative backtesting and utility tools with domain-specific schemas (EvidenceSpec, EvidenceDataset, BacktestResult). Tool names follow verb_noun convention (run_backtest, load_spec, dump_result). However, descriptions are terse (avg ~60 chars, well below the 194-char baseline), parameter descriptions are minimal or absent, and input/output schemas are not formally documented in the MCP registration layer. Tools reference complex custom types (EvidenceSpec, BacktestConfig) without inline schema definitions. No error handling guidance, no pagination for batch results, and no tool annotations (readOnlyHint/destructiveHint). The codebase is well-structured and domain-coherent, but the MCP interface lacks the LLM-optimization and schema rigor required for production agent use.
Convert a Python object to canonical string representation
Convert a Python object to canonical UTF-8 bytes for hashing
Compute portfolio-level metrics by combining multiple backtest results
Dump a BacktestResult to a JSON file
Load an EvidenceDataset from a JSON file
Load arbitrary JSON from a file with duplicate-key detection
Load an EvidenceSpec from a JSON file
Parameter descriptions are minimal or absent. Complex types like EvidenceSpec, BacktestConfig, and TickPosition are referenced but not explained inline. LLMs cannot infer what fields these objects require or how to construct them.
Tool descriptions are terse (avg ~60 chars vs 194-char baseline). Descriptions lack context on WHEN to use each tool, what it returns, and how it differs from similar tools. E.g., run_sensitivity_analysis and run_pbo_analysis both sweep parameters but their distinction is unclear.
No input/output schemas formally documented in MCP registration. Tools reference custom Pydantic models (BacktestResult, EvidenceDataset) but the MCP layer does not expose their JSON Schema definitions. Agents cannot validate inputs or plan downstream tool calls.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 45 | 2026-07-28+ | v2 |
Convert a BacktestResult to canonical JSON string representation
Run a single backtest with a spec and dataset, returning a BacktestResult with metrics and result_hash
Run multiple backtests in batch, returning a BatchResult with trial ledger and aggregate metrics
Run Probability of Backtest Overfitting (PBO) analysis via combinatorial walk-forward
Run a 7x7 sensitivity sweep over entry_alpha_gte and sizing_value, computing Sharpe and drawdown fan for each cell
Run a walk-forward backtest with anchored or rolling windows, returning aggregate metrics across folds
Compute SHA256 hash of the canonical representation of an object
Single-bar paper evaluation for crypto markets, returning fill and position updates
Verify a hash-linked deployment record chain for integrity and consistency
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). All tools are marked READ_ONLY or WRITE but MCP clients cannot detect this from the tool definition. Agents cannot reason about side effects or retry safety.
No error handling guidance. Tools do not document what errors can occur, whether they are retryable, or what the LLM should do next. E.g., run_backtest may fail if dataset is malformed, but no recovery path is documented.