Loading ToolBench

Reading the current corpus…

black-vault MCP server · D · ToolBench

black-vault

github.com/venkat22022202/black-vault

A hardened credential proxy and MCP server that gives AI agents and tools governed access to vaulted API keys with real-time cost tracking, budget caps, rate limiting, and audit trails. Supports universal OpenAI-compatible chat completion endpoint, MCP credential broker, and HTTP egress firewall.

2026-07-28+STREAMABLE_HTTP2 tools

56/100

Definition Quality68

Protocol Readiness57

Supportability36

Spec posture

Inferred effective spec 2026-07-28+·current baseline 2026-07-28

Static source inference · medium confidence · detected: stateful session

Deprecated protocol patterns detected

Summary

BlackVault exposes 2 tools via HTTP MCP endpoint. Both tools have descriptions and input schemas present, but descriptions lack context for when/why LLMs should call them, parameter descriptions are minimal or missing, and output schemas are not documented. The 'chat' tool has a well-structured input schema with enums and types, but the 'list_models' tool is trivial (no parameters). Error handling is present but generic. Overall structure is functional but falls short of production-grade documentation standards.

Tools (2)

chatwriteauthsource verified73/100

Run a chat completion on any allowed model (gpt-*, claude-*, gemini-*, open-source via Nebius). BlackVault injects the real provider key, enforces budget/rate/model limits, and audits the call.

list_modelsread onlyauthsource verified63/100

List the AI models this BlackVault token is allowed to use (across all vaulted provider keys).

Issues (6)

HIGHtool ↗

Output schemas not documented. 'list_models' and 'chat' tool descriptions do not specify what fields/structure the response contains. LLMs cannot plan downstream steps or extract required data without knowing the response shape.

list_modelschat

HIGHtool-description ↗

Parameter descriptions incomplete or missing. 'messages' parameter in 'chat' tool has a schema description but lacks domain context: 'OpenAI-style chat messages' does not explain what role values mean in BlackVault's context, whether system prompts are supported differently, or constraints on message length/count.

chat

MEDIUMpaginated-result ↗

No pagination or result limiting documented for 'list_models'. If many models are available, a large unfiltered response could exhaust context. No indication of whether results are paginated, capped, or how to filter by provider.

list_models

MEDIUMrecovery-guide ↗

Recommendations

Add explicit output schema documentation to 'list_models' and 'chat' tool descriptions. E.g. 'Returns: {models: [{id, name, provider, costPerMtkToken}], totalCount: number}' and 'Returns: {choices: [{message: {role, content}, finishReason}], usage: {promptTokens, completionTokens}}'. This allows LLMs to plan chaining.
Expand 'messages' parameter description to include: 'Array of OpenAI-format messages. Each message has role (system/user/assistant) and content (string). System messages set context; user messages are prompts; assistant messages are prior responses. Max 100 messages per call.' This gives domain-specific context.
Document 'max_tokens' and 'temperature' constraints in descriptions: 'max_tokens: integer, 1 - 4096, default 2048. temperature: float, 0.0 - 2.0, default 1.0. Higher temperature = more creative; lower = more deterministic.'
Add error classification to 'chat' tool description: 'On error, returns {error, errorCode, retryable, recoveryHint}. E.g. model not found → suggests listing available models; rate limit → ask user to wait; auth failed → check vault token.'
Mark 'list_models' with readOnlyHint=true in tool definition. Mark 'chat' with destructiveHint=false (it is read-only too, unless prompts have side effects). Document idempotency: 'Repeated calls with identical model + messages produce identical output; safe to retry.'
Clarify 'list_models' description: 'Lists all AI models this user is authorized to use across all vaulted provider keys (OpenAI, Anthropic, Google, etc.). Call this first to discover available models before invoking chat(). Returns model ID, name, provider, and cost-per-token baseline.'

Spec posture evidence

Inferred effective spec: 2026-07-28+.

Relies on Stateful initialize / Mcp-Session-Id (removed; protocol is stateless) - make each request self-contained

Score history

Overall score trend

First recorded score · v2 rubric

56/100

Scored	Grade	Overall	Spec posture	Rubric
2026-09-23	D	56	2026-07-28+	v2