Loading ToolBench
MCP server for JSON file operations with dot-notation path support
JsonMcpTool has three well-structured tools with clear input schemas and descriptions, but falls short of production-grade quality in several dimensions. All tools include verb-noun naming (get, set, remove) and parameter descriptions with types. However, descriptions are bare-minimum (80-120 chars), lack context about when to use each tool, and provide no guidance on error recovery. Output schemas are entirely undocumented, LLMs cannot predict what fields to expect from responses. No idempotency guidance, no pagination for large results, and no examples of field references for tool chaining. Error handling returns raw error objects rather than recovery suggestions. Security considerations like permission gates and audit trails are absent. The three tools are focused on JSON manipulation and avoid combining concerns, which is positive.
Get the value at the specified key path from a JSON file. Returns the value regardless of whether it's a string or an object.
Remove a key at the specified path from a JSON file. Works regardless of whether the value is a string or an object.
Set a string value at the specified key path in a JSON file. Creates nested objects as needed if the path doesn't exist.
No output schema documentation. LLMs cannot predict what fields the tools return (e.g., does 'set' return just a confirmation string, or structured {success: bool, key_path: string, value: string}?). This forces agents to guess downstream field names for tool chaining.
Error responses are unstructured strings ('❌ Error: ...') without recovery guidance. Errors should categorize failures as retryable, user-fixable, or fatal, and suggest next steps (e.g., 'File not found. Check the path or create the file first.').
Descriptions lack usage context. 'Set a string value...' does not indicate WHEN to use this vs. other JSON tools, what prerequisites exist (e.g., file must exist for 'set'), or what side effects occur (e.g., file is written to disk). Descriptions should answer: What does it do? When should I call it?
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 67 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 49 | - | v1 |
Tool 'set' claims to 'create nested objects as needed' but description does not clarify: will it overwrite existing values? Is it idempotent? Can you safely retry? Agents need explicit idempotency hints to avoid duplicate side effects.
Parameter 'key_path' uses dot notation (e.g., 'dashboard.title') but this is not a constrained format in the schema. No regex pattern, length limits, or character restrictions are declared. If invalid paths like 'dashboard..title' or '/' are passed, the tool fails with a cryptic error.
'remove' tool's description says it removes a key but does not specify: what is the return value? Is it the removed value (for undo capability), or just a confirmation? The code shows it returns the removed value as JSON, but the description omits this critical detail.
No pagination or result limiting. If a JSON file contains a large nested object, 'get' with an empty key_path returns the entire file, potentially thousands of lines. This can exhaust LLM context. Should cap results and offer pagination.
No permission gates or audit logging. Tools can read/write/delete any file the process has access to. No mention of permission checks, logging who called what, timestamps, or error traces for compliance/debugging.
Tool 'set' accepts only string values ('mcp.WithString("value", ...)'), but the description says 'Set a string value', yet JSON supports objects, arrays, numbers, booleans, null. This limitation is undocumented and may confuse users expecting full JSON type support.