Loading ToolBench
AI account management, token usage tracking, and budget enforcement
The server defines 9 tools with complete JSON schemas and descriptions. All tools follow verb_noun naming conventions (create_account, list_accounts, etc.). Descriptions are present for all tools (range: 40-290 chars) and generally explain what the tool does, though some lack WHEN to use them or dependency hints. All required parameters are documented with types and descriptions. However, there are moderate gaps: (1) output schemas are not documented in the provided code, (2) some parameter descriptions lack format/constraint detail (e.g., 'config' is a JSON string but no parsing guidance), (3) error handling guidance is absent, (4) no discussion of idempotency or retryability for stateful operations like report_usage. Baselines: tool descriptions average 194 chars (observed range is tight around 100-150); all 9 tools have descriptions (94% baseline met). Parameter descriptions are present but often generic.
Check budget status for an account (ok, warning, or blocked)
Connect a Claude Code account via OAuth. Call without code to start the flow (opens browser), then call again with the authorization code to complete.
Create a new AI provider account with auth config and optional budget
Get detailed account information including usage and config
Get environment variables needed to spawn a process with the account's credentials
List all AI accounts with usage stats and masked secrets
Output schemas not documented. No specification of what fields each tool returns or their types. LLMs cannot plan downstream calls or extract the right data.
Error handling guidance absent. No tool specifies what errors can occur, whether they are retryable, or what the LLM should do next. E.g., remove_account should state if the account_id is not found, return a helpful message like 'Account not found. Try list_accounts() to see available accounts.'
Parameter 'config' in create_account is described as 'JSON string with auth config' but lacks detail on format, required fields, or parsing rules. LLMs will struggle to construct valid JSON. Example: should it be escaped? Are there required vs optional fields per provider?
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 58 | 2026-07-28+ | v2 |
| 2026-03-09 | D | 57 | - | v1 |
Remove an AI provider account
Record a usage event and update account running totals
Set the budget limit and alert threshold for an account
get_account_env description is vague: 'Get environment variables needed to spawn a process'. Does it return a dict? A shell script? A .env file string? Unsafe for LLMs without explicit format specification.
No documentation of idempotency or retryability. report_usage appears to record a usage event, should the agent be able to retry without double-counting? Should create_account with the same name be idempotent or fail with 'already exists'?
connect_claude_code_account has a multi-step OAuth flow (call once to start, call again with code). No explicit documentation of the state machine: what fields does the first response return? What is the 'state' parameter used for? This is a critical multi-round-trip pattern that needs clear documentation.
Numeric parameter constraints missing. alert_at is described as 'percentage 0-100' but no explicit min/max in description. budget_usd can be 0 for unlimited, this non-obvious default should be called out. LLMs cannot see JSON Schema constraints; they depend on text descriptions.
Pagination and result limits not addressed. list_accounts returns 'all AI accounts', if there are 1000 accounts, the response could explode the context window. No mention of limit/offset or total count in the schema.