Loading ToolBench
MCP server for Splunkbase
Three tools with basic descriptions and partial schemas. Tool names follow verb_noun convention (search, get_app_latest_version, download_app). Descriptions exist but are brief (10-60 chars, below the 50-200 char baseline for LLM optimization). No output schemas documented. Error handling present in code (ctx.error calls) but not surfaced in tool definitions. Security concern: credentials injected via environment but no mention in tool descriptions. No tool annotations (readOnlyHint/destructiveHint) despite clear risk profiles.
Download a specific version of an app. If no version is specified, downloads the latest.
Get the latest compatible version of an app for a specific Splunk version.
Search Splunkbase for apps.
Parameter descriptions missing. Input schemas define types (string, integer, boolean) but lack descriptions explaining what each parameter controls. LLMs cannot infer parameter meaning from names alone, 'app' could be ID or name, 'output_dir' lacks format guidance.
No output schemas documented. Tools return strings, dicts, or lists but LLMs have no schema to parse results. search() returns a list, get_app_latest_version() returns Dict[str, Any], download_app() returns str, none are formally specified.
Tool descriptions too brief. 'Search Splunkbase for apps' (27 chars) and 'Download a specific version of an app' (37 chars) lack context on when to use, prerequisites, or what to expect. Baseline is 50-200 chars for LLM optimization.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 46 | 2026-07-28+ | v2 |
No tool annotations. download_app is destructive (WRITE risk) but lacks destructiveHint annotation. search and get_app_latest_version are read-only but lack readOnlyHint. Agents need this metadata to reason about safety and retry logic.
Error handling not surfaced in tool definitions. Code includes ctx.error() calls and exception handling, but tool descriptions do not document error cases, recovery steps, or what the LLM should do if a call fails (retry, ask user, abort).