Loading ToolBench
Self-hosted MCP gateway: one streamable-HTTP endpoint federating many MCP servers with namespaced tools, per-tool toggles, roles, and secure upstream credential injection
MSPStack Gateway presents well-structured self-management tools with clear naming conventions (gw_* prefix), comprehensive schemas, and detailed descriptions. All 12 tools have explicit input schemas with proper type definitions and descriptions. Tool names follow verb_noun patterns (list, set, install, remove, refresh, backup). Descriptions average ~120 chars and explain WHAT the tool does and WHEN to use it. However, output schemas are not documented in the source code provided, responses are inferred from context. Error handling guidance is minimal; tools lack recovery hints for common failure modes. Security-sensitive operations (gw_remove_server, gw_set_grant) have confirmation parameters but no explicit permission-gate documentation. Composition is strong: tools chain naturally (list_tools → set_tools_enabled → set_tool_tier). Parameter naming is consistent and unambiguous (upstreamId, roleName, maxTier). No secrets exposed as parameters, credential injection is server-side via preset parameters using references (kv:/bao:/${ENV}).
Take a database snapshot immediately (VACUUM INTO), prune to the retention limit, and ship it off-instance when configured.
Install (or re-install) an upstream from a preset. Secret parameters must be REFERENCES (kv:/bao:/${ENV}) — never a raw secret, which would end up stored in the spec.
Installable server presets: id, title, description, the parameters each one needs, and its recommended grants.
Configured upstream MCP servers with their mode flags. Header/env values are redacted — only keys and whether the value is a secret ref, an env ref, or a literal.
Tools in the catalog with their effective tier, group and enabled state. Filter by upstream, tier, group, enabled state, or a name substring.
Re-read every upstream's tool list now (also happens automatically on list_changed and after reconnects).
Output schemas not documented in source. Tool descriptions explain inputs but do not specify return types, field names, or structure. LLMs cannot plan downstream tool calls or extract chaining IDs without documented outputs.
Error handling lacks recovery guidance. Tools do not document what errors are retryable, user-fixable, or fatal. No examples of error messages or suggested next steps for common failures (e.g., upstream not found, permission denied).
Destructive operations (gw_remove_server) require a 'confirm' boolean but lack dry-run support. No confirmation-request pattern for multi-step operations. Agents cannot preview changes before committing.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 66 | <=2025-11-25 | v2 |
Remove an upstream and everything attached to it: tool settings, per-role grants and overrides. Personal credentials for it are orphaned, not deleted.
Set a role's ceiling for one upstream by role NAME: none / read / write / destructive. 'none' closes the upstream for that role.
Enable or disable a whole upstream server (its tools disappear from every session while disabled).
Override a tool's tier (read/write/destructive), or clear the override to fall back to the annotation-derived tier. Raising the tier is how a read-only tool that hands out secrets is kept away from low roles.
Enable or disable tools for EVERYONE (the admin envelope, not a personal preference). Scope with tier and/or group, a single toolName, or neither for the whole upstream.
Gateway health: version, connected upstreams with tool counts and last errors, catalog size, secret-store scheme, auth mode.
Permission-gate documentation missing. Tools like gw_set_grant and gw_remove_server do not declare required permissions (e.g., 'admin:upstream', 'write:policy'). Unclear what authorization checks apply.
Parameter dependencies undocumented. gw_set_tools_enabled accepts tier, group, and toolName as optional filters but does not explain how they combine (AND vs OR logic). gw_install_preset params field is untyped (additionalProperties: string), no schema for individual preset parameters.