Loading ToolBench
Read-only access to Secureframe compliance platform data
The server provides 11 read-only tools with consistent naming conventions (all start with 'list_') and complete parameter schemas. However, descriptions are generic and often under-differentiated across similar tools. Parameter descriptions are present but minimal. Output schemas are not documented. Error handling is basic (generic API error wrapper with no recovery guidance). This is a typical C-grade community server, functional but lacking the polish and LLM-optimization of production tools.
List security controls with filtering support
List devices with filtering support
List compliance frameworks with filtering support
List integration connections with filtering support
List repositories with filtering support
List framework scopes for a repository
Descriptions lack differentiation and context. All 11 tools use generic phrases like 'with filtering support' without explaining WHAT each resource type is, WHEN to call it vs. similar tools, or what unique data it provides. 'List security controls with filtering support' tells an LLM nothing about whether to use this instead of list_tests or list_frameworks.
Output schema is not documented. The code returns response.json() from the Secureframe API, but LLMs cannot see what fields the response contains, their types, or how to chain them to downstream tools. Without documented output schema, agents must guess at the structure (pattern:response-shaper).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 58 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 37 | - | v1 |
List compliance tests with filtering support
List TPRM vendors with filtering support
List user accounts with filtering support
List users with filtering support
List vendors (legacy) with filtering support
Error handling returns a generic {'error': '<message>'} dict with no actionable guidance. If a Lucene search fails or pagination is out of bounds, the LLM receives a raw API error and has no direction on recovery (e.g., 'Invalid query syntax. Try simple words without special characters' or 'Page out of range. Available pages: 1 - 5'). Errors should guide recovery (pattern:recovery-guide).
API credentials are stored in environment variables and injected at startup, which is correct. However, no documentation on the server's README or in code comments explains the required SECUREFRAME_API_KEY and SECUREFRAME_API_SECRET format, scope, or how to obtain them. Users cannot self-serve setup.
Pagination defaults to per_page=100 with no documented upper limit or guidance on when results exceed context window. If a user has 10,000 controls, requesting page 1 with per_page=100 still returns 100 items, no indication of total count, truncation, or recommendation to use smaller page sizes. Response should include total count and offer pagination hints (pattern:paginated-result).
No permission checks or audit logging. Tools are read-only, so the risk is lower, but there is no mechanism to trace which agent/user called which tool, when, with what parameters. For compliance-sensitive data (users, devices, vendor info), audit trails are often required (pattern:audit-trail).
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). While all tools are read-only, fastmcp does not expose a way to annotate them. LLMs benefit from explicit hints that these tools are safe to call repeatedly and cannot modify state.
Parameter 'search_query' accepts Lucene syntax but the description says 'Lucene search query' without explaining what fields are searchable, what operators are supported (AND, OR, NOT, wildcards?), or common errors. An LLM may pass 'status:active AND created:2024' without knowing whether those fields exist. Description should include a format hint or link to docs (pattern:constrained-input).