Loading ToolBench
AI-agent workspace for quantitative trading with MCP tools for strategy backtesting, dataset management, and paper trading
QuantDesk demonstrates solid tool design with complete input schemas and descriptions for all 12 tools. Naming follows verb_noun conventions (register_dataset, run_backtest, record_run_metrics). Descriptions are substantive (100-300+ chars) and explain WHEN to use each tool and what it returns. However, several tools lack output schema documentation, parameter descriptions are sometimes generic, and error handling guidance is minimal. Tool composition is strong, tools chain logically (run_backtest → record_run_metrics → request_validation). Security considerations are present (Risk Manager gating, consent requirements documented). The server targets a specialized domain (quantitative trading) with domain-specific terminology that may confuse generic LLMs without additional context.
Mark the current experiment finished. Requires prior user consent.
Read current paper trading state. Returns the active session with live PnL/positions if running, or the latest historical session with active: false. Use this whenever you answer the user about paper trading — never guess from memory, the container may have stopped since your last turn. No consent needed.
Promote a completed backtest run to paper trading. Risk Manager approval is recommended but not enforced; if the run was rejected, surface that explicitly to the user and obtain consent before calling. Starts a dry-run container on the desk's engine. One session per desk. Requires prior user consent. Paper trading is NOT supported for generic-engine desks.
Close this experiment and open a new one. Requires prior user consent.
Mandatory after every run_backtest. Publish the metrics that describe this strategy on the run: use rawStats from the backtest response as a starting point, but ADD / REPLACE with strategy-specific ones (RPI, inventory deviation, volume/liquidity ratio, sharpe, hit-rate-at-N-bars, whatever the strategy's thesis actually hinges on). Merge-by-key: new keys appended, same-key entries override. replace: true discards whatever was there (use when re-writing metrics from scratch). Prefer runNumber — UUIDs are not in your prompt. Total cap is 12 metrics per run. Safe to call repeatedly as you refine; publishes a run.status event so the UI refreshes. No consent needed.
Output schemas not documented. Tools like run_backtest, get_paper_status, and run_script lack explicit return type documentation. LLMs cannot plan downstream calls or extract required fields without knowing response structure.
Domain-specific terminology without LLM guidance. Terms like 'rawStats', 'RPI', 'inventory deviation', 'sharpe', 'hit-rate-at-N-bars' appear in descriptions but lack definitions. LLMs unfamiliar with quantitative trading may misinterpret or misuse these concepts.
Minimal error handling guidance. Tool descriptions state what happens on success but do not explain failure modes, retryability, or recovery steps. E.g., run_backtest does not document what happens if no dataset is registered or if the strategy crashes.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 68 | 2026-07-28+ | v2 |
Register a dataset that already exists on disk (e.g. produced by your fetcher script via run_script) and link it to the current desk. MUST be called before run_backtest after you fetch data yourself. Pass tradingMode so the catalog can distinguish spot from futures — otherwise a futures dataset would collide with a spot row that has the same pair / timeframe / window.
Dispatch Risk Manager validation on a specific run. You almost never call this directly. Iteration reviews are auto-dispatched by run_backtest; this tool exists only for the user-initiated path — if the user clicks the Validate button in the Runs table (or types "Validate Run #N"), you'll see a "validate" comment from them and you forward it to the RM via this tool. Prefer runNumber — UUIDs are not exposed in your prompt. Call once, then end your turn. The Risk Manager runs asynchronously; you will be retriggered with the verdict. No consent needed (it's a user-initiated action).
Execute the strategy. Requires at least one registered dataset. Returns {runId, runNumber, rawStats:{returnPct,drawdownPct,winRatePct,totalTrades}, autoDispatched?, message}. The engine no longer publishes display metrics by default. rawStats is the engine's universal summary; you must decide which of those + which strategy-specific measurements belong on the run card and publish them with record_run_metrics before ending your turn. Non-baseline runs: run_backtest AUTO-DISPATCHES the Risk Manager — do NOT call request_validation and do NOT analyse metrics in detail (that is the RM's job). Call record_run_metrics immediately (so the RM sees your framing of the result), then end your turn. Baseline run: no RM review, no auto-dispatch; publish metrics, analyse the result, and plan iteration 1.
Execute an agent-authored script in the generic sandbox container. Use for fetchers, exploration, any script — NOT the final backtest (use run_backtest for that). Available on all desks. No consent needed.
Rename the current experiment. No-op for Experiment #1. No consent needed.
Stop the active paper trading session. No retrigger. No consent needed.
Risk Manager only: attach the verdict to the run being reviewed.
Consent requirements documented in descriptions but not enforced via tool parameters. Tools like new_experiment, complete_experiment, and go_paper state 'Requires prior user consent' but lack a consent_token or confirmation parameter. This shifts enforcement burden to the agent.
Parameter descriptions lack format/constraint details. 'configFile' and 'entrypoint' in run_backtest do not specify expected file extensions, paths (relative vs absolute), or validation rules. LLMs may pass invalid paths.