Loading ToolBench
Conifer gateway as an MCP server over stdio: the catalog as tools an agent can call mid-task. Provides tools for model selection, cost-aware completions, model comparison, embeddings, and balance checking.
Five tools with complete schemas and strong descriptions (avg 180 chars). All parameters typed and described. Tool names follow verb_noun pattern (list_, choose_, complete, compare, embed). Output schemas documented in descriptions. Key strength: cost transparency and capability filtering. Weakness: no error handling guidance, no recovery hints, no confirmation patterns for destructive operations (conifer_complete, conifer_compare, conifer_embed all modify state but lack dry-run or confirmation support).
Pick the cheapest catalog model that DECLARES the capabilities you need. This is a catalog read, not a second router: it ranks what the catalog states, skips models with undeclared capabilities rather than assuming them, and skips unpriced models rather than assuming they are free.
Run the SAME prompt across several models in parallel and return each answer beside its exact settled cost, cheapest first. This is how you pick a model for a pipeline step with evidence instead of a guess: try the candidates on a real example from the job, read the answers, read the costs. A model that fails reports its error in place — one bad model never sinks the comparison. Each model's call is its own billed turn; max_cost_nanousd caps EACH turn, not the total.
Ask any model on the gateway one question — or hand it a whole conversation — and get the answer back with the exact settled cost of that call in nanodollars ($1 = 1e9). Use it mid-task to consult a different model than the one driving this session: a cheap one for bulk work, a reasoning one for a design question. Set max_cost_nanousd and the gateway refuses the call before spending anything if it could cost more.
Turn text into embedding vectors, with the exact settled cost of that call. Send one string or a batch; you get one vector per input, in the order you sent them. Use it to build or query a semantic index mid-task, or to rank candidates by cosine similarity (this gateway serves no rerank door, and embeddings are the honest substitute). Only models whose caps include "embeddings" answer here — call conifer_choose_model with caps:["embeddings"] to pick the cheapest one.
No error recovery guidance. Tools lack descriptions of failure modes and recovery steps. E.g., conifer_complete does not document what happens if a model is unavailable, rate-limited, or returns an error. LLMs cannot self-correct without explicit guidance.
No confirmation or dry-run pattern for state-modifying tools. conifer_complete, conifer_compare, and conifer_embed all incur costs and modify external state (billing, API calls). No dry-run mode or confirmation step to prevent accidental expensive calls.
max_cost_nanousd parameter lacks validation guidance. Description does not explain what happens if the call would exceed the limit (does gateway refuse before spending? does it return a cost estimate?). LLMs cannot reason about cost safety without explicit behavior documentation.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 66 | 2026-07-28+ | v2 |
List the models this Conifer key can call, with declared capabilities, context windows, and the AS-CHARGED price for each entry's lane. Absent fields mean the catalog did not declare them, not that the model lacks the feature.
conifer_compare returns per-model errors but does not document error structure or how LLMs should interpret partial failures. If 3 of 5 models fail, what does the response look like? How should the agent decide whether to retry or proceed with partial results?
Output schemas not formally documented in code. Descriptions mention what is returned (e.g., 'answer back with the exact settled cost in nanodollars') but no structured schema definition visible for tool responses. LLMs cannot reliably parse unstructured output.