Loading ToolBench
Walk-Forward Efficiency, parameter-stability, and lock-after-optimization auditing for algo/quant strategy robustness testing
Four tools with complete schemas and strong domain-specific descriptions. All tools follow verb_noun naming (walk_forward_efficiency, parameter_stability_score, lock_after_wfo_check, walk_forward_window_generator). Descriptions are detailed (150-250 chars) and explain WHEN to use each tool and what it reveals. Input schemas use Zod with proper type definitions and descriptions. However, output schemas are documented only in code comments/return statements, not in formal schema declarations. No error handling guidance for invalid inputs (e.g., negative metrics, malformed dates). No tool annotations (readOnlyHint, destructiveHint). All tools are READ_ONLY by design but this is not formally declared.
Audit a parameter change log against a stated walk-forward-optimization lock date. Flags any parameter edits made AFTER optimization concluded — hand-tuning post-lock quietly reopens the exact overfitting risk that walk-forward testing was supposed to have already ruled out, and it's invisible unless someone checks the dates.
Score how fragile a chosen parameter combination is by comparing its performance to nearby combinations in the search neighborhood (e.g. one step off in each optimized dimension). Distinguishes a robust plateau (nearby combos also perform well) from a fragile spike (an isolated peak surrounded by mediocrity) — the second is a classic curve-fit signature that a single backtest number can't reveal.
Compute Walk-Forward Efficiency (WFE) — the ratio of out-of-sample to in-sample performance across your actual walk-forward windows. This is the standard acceptance test for whether a strategy's edge survives re-optimization over time, not just a single train/test split. Industry rule of thumb: WFE >= 0.5 is considered robust; well below that means the strategy is likely curve-fit to each in-sample window.
Generate walk-forward optimization window boundaries (in-sample/out-of-sample index ranges) for either rolling (fixed-size in-sample window slides forward) or anchored (in-sample start stays fixed, window grows) walk-forward testing — a practical setup tool for structuring a WFO run correctly before you have any results to validate.
No formal output schema documentation. Return types are inferred from code (JSON objects with typed fields) but not declared in tool registration. LLMs cannot reliably predict response structure.
No error handling guidance. Tools accept numeric and date inputs but do not document validation rules (e.g., 'in_sample_metric must be non-negative', 'wfo_lock_date must be ISO 8601'). Invalid inputs will fail silently or with generic errors.
No tool annotations. All four tools are read-only (no side effects) but lack readOnlyHint annotation. This forces LLMs to infer safety from descriptions rather than machine-readable metadata.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 69 | 2026-07-28+ | v2 |
Parameter 'mode' in walk_forward_window_generator uses enum but lacks examples in description. LLMs may not understand the practical difference between 'rolling' and 'anchored' modes without concrete explanation.
Date parameter in lock_after_wfo_check (wfo_lock_date) requires ISO 8601 format but does not validate or provide recovery guidance if format is wrong. No pattern constraint or example in schema.