Loading ToolBench
Live pricing, context windows and identifiers for every major LLM API. Stop hardcoding a model table that goes stale.
Statlyte has clear, well-named tools with good descriptions (avg 150 chars, within baseline 194). All 5 tools start with action verbs (list_, get_, estimate_, cheapest_, scheduled_). Input schemas are present and typed. However, parameter descriptions are sparse or missing entirely, most numeric params lack range/constraint guidance. Output schemas are not documented. Error handling is minimal (generic 'unknown model' responses). No tool annotations (readOnlyHint, etc.). The server is read-only and stateless, which is good, but lacks the depth expected for production use.
Rank every generally-available model by what a monthly token volume would cost. Use to answer "what is the cheapest model for this workload".
Cost in USD of a call or workload against one model, given raw token counts.
Current published pricing and specs for one model. Accepts a statlyte id (anthropic/claude-opus-5) or a vendor API id (gpt-5-mini).
List LLM models with current published pricing (USD per million tokens, or nonTokenPrice for audio/image models billed per minute/character/etc), context windows and API identifiers. Optionally filter by provider.
Price changes vendors have announced for a future date, with the old and new figures. Check before committing to a model.
Numeric parameters lack range constraints and validation guidance. 'inputPerMonth', 'outputPerMonth', 'limit', 'input', 'output', 'cacheRead' have no min/max or format hints in descriptions. LLMs may pass absurd values (negative tokens, 1e9 tokens) that break cost calculations.
Output schemas are not documented. Callers cannot predict response structure (e.g., does 'models' array contain 'prices' as object or array? Is 'monthlyUsd' always present?). LLMs must infer structure from examples, risking parsing errors.
Error handling is generic and non-actionable. 'unknown model' error does not guide recovery (e.g., 'Try list_models() to see available models'). No error categorization (retryable vs. fatal). Agents cannot self-correct.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 66 | 2026-07-28+ | v2 |
No tool annotations. All tools are read-only and idempotent, but lack readOnlyHint and idempotentHint metadata. Agents cannot infer safety properties without explicit hints.
'model' parameter in get_model_pricing and estimate_cost accepts both statlyte IDs and vendor API IDs but does not document the format or provide examples. LLMs may guess wrong (e.g., pass 'claude' instead of 'anthropic/claude-opus-5').