Loading ToolBench
A rule-based data validation library server exposing validation-lib with support for rule discovery, entity validation, workflow management, and loan data handling
This server has 26 tools with mixed quality. Strengths: most tools have descriptions (though often generic), clear single responsibilities, and action verbs in names (validate, batch_validate, discover, list, etc.). Weaknesses: input parameter schemas are visible but descriptions for many parameters are minimal or absent; output schemas are undocumented; error handling is not evident in code; no tool annotations (readOnlyHint/destructiveHint) despite having tools marked with risk levels in metadata; composition is reasonable but some tools combine multiple concerns (e.g., reload_logic + cache management). The server has rigid workflow routing rules baked into instructions rather than expressed as tool constraints. Naming is generally solid (verb_noun pattern) but parameter naming could be more explicit (e.g., 'folder' alone vs 'folder_name'). Many tools lack actionable error recovery guidance.
Validate multiple entities in a single call.
Validate all files in the inbox folder.
Validate a batch of loan files from a folder.
Permanently delete all files in a folder.
Convert nested physical loan dict to flat logical form.
Convert flat logical dict to physical form.
Permanently delete loan files.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract results effectively. Tools return lists or dicts with unstated fields.
Parameter descriptions are sparse or generic. Many parameters lack actionable guidance on format, constraints, or dependencies. E.g., 'Entity dict with a $schema field' does not explain the required structure.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 49 | 2026-07-28+ | v2 |
List all rules for an entity type and ruleset with their metadata.
List all available ruleset names.
Edit loan fields in a workflow file.
Table view of all folders (or one) with timestamps and principal amounts.
Get instructions for generating a realistic test loan.
Check how old the local rule cache is.
Get loan history and notes.
Return a formatted command reference grouped by category. Call when the user asks for help.
Browse the cached rule file tree.
Move loans between folders.
Live auto-refreshing counts panel (use only when user says 'quick' or 'short').
Read a rule or schema source file.
Read specific files, or omit to read all.
Refresh the inbox folder (handles everything server-side).
Force a fresh fetch of all rules from GitHub.
Find loans matching a field value.
Validate a single entity against a named ruleset.
Dry-run summary for a whole folder or specific files (no notes written).
Write a file to the workflow directory.
No error handling or recovery guidance in tool implementations visible. Destructive tools (delete_workflow_files, clear_workflow_folder) do not support dry-run, confirmation, or rollback. Code shows bare ToolError exceptions without actionable context.
Tool annotations missing. Metadata explicitly marks tools with Risk levels (READ_ONLY, WRITE, REVERSIBLE, DESTRUCTIVE) but these are NOT expressed as readOnlyHint/destructiveHint in the MCP schema. LLMs cannot reason about side effects.
Rigid workflow routing logic embedded in server instructions rather than expressed as tool constraints. Instructions say 'NEVER call full_workflow_summary() when user says quick', this is a protocol concern that should be enforced by tool design, not by LLM adherence to natural language rules.
No pagination support for tools returning lists. batch_validate, discover_rules, list_logic_files, read_workflow_files, search_workflow have no offset/limit or cursor mechanism. Large result sets will blow context windows.
Ambiguous parameter naming. 'folder' alone does not clarify whether it accepts an internal name, path, or display name. 'relative_path' and 'relative_paths' invite path traversal if inputs are not sanitized. 'changes' dict (edit_loan_file) lacks schema specification.
No input validation hints in parameter descriptions. e.g., read_logic_file(relative_path) accepts arbitrary strings with no guidance on allowed characters, path depth, or security boundaries. No mention of directory traversal protection.