Loading ToolBench
Solo Founder OS agent #11 — monthly bill audit across Vercel / Anthropic / Supabase / OpenPanel / HyperDX. Flags underused SaaS + spend spikes.
Three tools with clear verb-noun naming (get_*, top_*) and reasonable descriptions (120-180 chars). All tools have input schemas with type definitions. However, parameter descriptions are minimal or absent: get_provider's 'name' param lacks enum constraint despite accepting only 6 known values; top_savings's 'n' param has no bounds or default behavior documented. Output schemas are undocumented, LLMs cannot predict return structure. Error handling is present but generic ('Unknown provider', 'failed'). No tool annotations (readOnlyHint, idempotentHint). Composition is sound: tools chain logically (get_monthly_report → top_savings). Naming is clear but could be more specific (e.g., 'get_cost_report' vs 'get_monthly_report').
Generate the cost-audit monthly report — the full markdown bill audit across Vercel, Anthropic, OpenPanel, HyperDX, Supabase, and GitHub Actions. Returns: markdown report with total $ MTD, sorted waste findings, per-provider breakdown.
Run a single billing provider and return its findings as markdown. Args: name: one of vercel, anthropic, openpanel, hyperdx, supabase, github_actions. Returns: markdown summary for that one provider.
Return the top N waste findings across all providers, sorted by estimated monthly $ savings. Args: n: how many findings to return (default 5). Returns: markdown list of top findings with provider tag + $ saving.
Parameter 'name' in get_provider accepts only 6 known values (vercel, anthropic, openpanel, hyperdx, supabase, github_actions) but is defined as free-form string with no enum constraint. LLMs will hallucinate invalid provider names.
Parameter 'n' in top_savings has no documented bounds, default behavior, or constraints. LLMs may pass negative, zero, or absurdly large values (e.g., 999999), causing performance issues or unexpected truncation.
Output schemas are completely undocumented. All three tools return markdown strings, but LLMs cannot predict field structure, making downstream parsing and chaining difficult. No documentation of what fields appear in the markdown or how to extract data.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 51 | 2026-07-28+ | v2 |
Error messages are generic and non-actionable. 'Unknown provider' and 'failed' do not guide the LLM on recovery steps. Should include available options or suggest alternative tools.
No tool annotations (readOnlyHint, idempotentHint, destructiveHint). All three tools are read-only and idempotent, but this is not declared in the schema, forcing LLMs to infer safety properties.