Loading ToolBench
Three independent MCP servers: embedded-recon (firmware analysis), security-audit (website security posture), and smb-digest (RSS/Atom feed digestion)
Three tools with clear security-focused purposes and reasonable descriptions (150-250 chars each). All tools have input schemas with type definitions. However, output schemas are documented only in prose within descriptions, not as formal JSON Schema. Parameter descriptions are present but minimal (20-40 chars). No enum constraints on parameters. Error handling guidance is generic ('On error: { ok: false, error, fix }') without actionable recovery steps. Tools follow single-responsibility principle but lack composition guidance (e.g., which tool to call first, how results chain). No tool annotations (readOnlyHint, idempotentHint) despite all being read-only operations.
Passively analyze a LOCAL firmware/binary file and report its security-relevant recon. Statically inspects a file at `path` — NO execution, unpacking, network, or modification. Reports size, Shannon entropy (encryption/compression signal), detected format signatures (ELF, gzip, ZIP, SquashFS, U-Boot uImage, Android boot, XZ, tar, JFFS2), and interesting strings (URLs, embedded key/cert headers, hardcoded-secret keywords, version numbers). Parameters: path: absolute path to a local firmware image, e.g. "/tmp/router-fw.bin". Returns: { ok, target, size_bytes, entropy, detected_formats[], strings{}, summary, findings: [ {severity, title, detail, fix} ] }. On error: { ok: false, error, fix }. Do NOT use to analyze firmware you are not authorized to inspect. This is triage, not full RE.
Audit the security posture of a PUBLIC website by URL (passive, read-only). Fetches only public HTTP response headers and the presented TLS certificate. Performs NO scanning, exploitation, authentication, brute-forcing, or crawling. Answers "is this site configured securely?" for a site the user owns or is authorized to assess. Parameters: url: A public http(s) URL, e.g. "https://example.com". Internal/loopback/private and cloud-metadata addresses are refused (SSRF guard). Returns a JSON object: { ok, target, status, summary, findings: [ { id, severity: "high"|"medium"|"low"|"info", title, detail, fix } ] } On refusal/failure: { ok: false, target, error, fix }. Do NOT use for penetration testing, authenticated areas, bulk/repeated scanning of third-party sites without authorization, or any active/intrusive probing.
Fetch and normalize public RSS/Atom feeds into a deduped, recent news digest. Use to monitor industry news, competitor blogs, or regulator feeds for a business. Returns clean structured items; YOU (the agent) write the summary/brief from them. Passive fetch only. Parameters: feed_urls: list of public RSS/Atom feed URLs (max 20). Internal/private hosts are refused. limit: max items to return (default 10). Returns: { ok, count, summary, items: [ {title, link, published, summary, source} ], errors[] }. On bad input: { ok: false, error, fix }. Do NOT use for: paywalled/authenticated content, scraping non-feed HTML, or high-frequency polling.
Output schemas documented only in prose descriptions, not as formal JSON Schema. LLMs cannot parse expected response structure from docstrings alone.
Parameter descriptions are minimal (20-40 chars). 'path' for analyze_firmware and 'url' for audit_url lack format/constraint details. LLMs cannot infer valid input ranges or formats.
No enum constraints on parameters. 'limit' in fetch_digest accepts any integer; no min/max bounds documented. 'feed_urls' accepts list but max 20 is only mentioned in description text, not schema.
Error handling is generic. 'On error: { ok: false, error, fix }' provides no actionable recovery guidance. LLMs cannot determine if errors are retryable, user-fixable, or fatal.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 49 | 2026-07-28+ | v2 |
No tool annotations despite all tools being read-only. Missing readOnlyHint and idempotentHint annotations prevent LLMs from optimizing retry logic and caching strategies.