Loading ToolBench
Local multi-account paper-trading CLI, live TUI dashboard, REST + MCP server
21 tools with complete schemas and descriptions, but quality is uneven. Naming is strong (verb_noun pattern, clear intent). Descriptions average ~120 chars, mostly adequate but some lack context on when/why to use. Parameters are well-typed with defaults, but output schemas are undocumented, LLMs cannot predict response structure. Error handling is minimal; most tools return raw strings without guidance. No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear read/write distinction. Idempotency keys present on write operations (good), but no confirmation pattern for destructive ops. Missing: output schema documentation, error recovery guidance, and per-tool risk classification.
Create an account. Reuse idempotency_key safely when retrying the same mutation.
List all paper accounts and their cash balances.
Buy shares. Market order fills immediately at the live price; pass limit for a resting limit order.
Buy option contracts (calls or puts). Limit orders are supported; market orders fill at the live mark.
Cancel a pending order by ID.
Deposit cash into an account.
Check if the market is currently open and when the next transition (open/close) occurs.
Output schemas undocumented. Tools return strings (e.g., 'ok', formatted text) but LLMs cannot predict structure. No documentation of what fields/format to expect from quote(), option_chain(), performance(), or backtest results.
No tool annotations. Write operations (buy, sell, deposit, withdraw, cancel_order) lack destructiveHint=true. Read operations lack readOnlyHint=true. Idempotent operations (with idempotency_key) lack idempotentHint=true. LLMs cannot infer safety/retry semantics.
Minimal error guidance. Tools return raw error strings (e.g., 'error: unknown error') without recovery hints. No categorization of retryable vs user-fixable vs fatal errors. LLMs cannot self-correct or know what to try next.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 58 | <=2025-11-25 | v2 |
Fetch the option chain for a symbol: all strikes and expirations with bid/ask/last prices.
List paginated order history, including source agent and idempotency key.
Portfolio performance since inception: return, CAGR, Sharpe, Sortino, volatility, max drawdown. Reconstructs a daily equity curve from the trade/cashflow ledger and real historical prices.
Mark-to-market P&L report for an account: positions, unrealized P&L, cash, total equity.
Backtest the account's current open positions and cash with backtesting.py. The universe is read from this account's SQLite positions, options without reliable continuous history are reported as skipped, and results include the equity curve, return, CAGR, volatility, Sharpe, Sortino, and drawdown. The default window is five years; request up to 36500 calendar days. This is a current-holdings retrospective, not an out-of-sample strategy test.
List open positions (symbol, side, qty, avg cost, asset class) for an account.
Get the current market price for a symbol (stock, ETF, crypto, future, or option).
Rename a paper portfolio. Updates positions, orders, history, and the default pointer.
Sell shares (selling more than held opens a short). Market order unless limit is given.
Sell option contracts (calls or puts). Selling without owning opens a short position.
One-line-per-portfolio snapshot for all accounts: cash, equity, unrealized P&L, position count. Fast overview for deciding which account to act on.
Check all pending limit orders against current prices and fill any that crossed.
Validate that a symbol exists and return its asset class (spot, future, option).
Withdraw cash from an account.
No confirmation pattern for destructive operations. buy(), sell(), cancel_order(), withdraw() can cause financial loss but lack dry-run or confirmation step. Agents may execute irreversible trades without user approval.
String-based responses instead of structured output. positions(), orders(), pnl(), summary() return formatted text. LLMs must parse unstructured output, risking extraction errors and wasting tokens. Should return JSON objects with typed fields.