Loading ToolBench
Parse and analyze mortgage documents (Loan Estimates, Closing Disclosures) using MISMO standards
This mortgage document parser has 4 tools with descriptions and declared input schemas, but exhibits significant quality gaps. Tool naming follows verb_noun convention adequately. However, descriptions lack depth and fail to guide LLM decision-making. Critical issues: (1) Schemas are minimal, only 'pdf_url' or basic object types without property-level documentation; (2) No documented output schemas; (3) Error handling not visible; (4) The 'hello' tool is a placeholder that shouldn't ship in production; (5) Composition issues: compare_documents accepts object parameters but doesn't specify what fields those objects must contain or what fields are returned. The server uses the 'mcp' framework (requirements.txt lists 'mcp>=1.0.0'), suggesting modern MCP SDK, but the source code samples do not show the actual tool registration implementation, limiting verification of schema completeness.
Compare Loan Estimate and Closing Disclosure for TRID compliance and cost changes
Basic tool for testing API connectivity and framework integration. More mortgage tools coming soon.
Parse a Closing Disclosure PDF into MISMO-compliant structured data
Parse a Loan Estimate PDF into MISMO-compliant structured data
Output schemas undocumented. parse_loan_estimate and parse_closing_disclosure return 'MISMO-compliant structured data' but no schema is specified. LLMs cannot plan downstream tool calls without knowing what fields to expect (e.g., does it return loan_amount, interest_rate, term_months?). compare_documents returns comparison results but structure is not documented.
Input parameter schemas are incomplete. compare_documents accepts 'loan_estimate' and 'closing_disclosure' as type 'object' with no property definitions. LLMs cannot validate structure or know which fields are required. What properties must these objects have? What are their types and constraints?
Descriptions are too brief and lack actionable guidance. 'Parse a Loan Estimate PDF into MISMO-compliant structured data' (59 chars) does not explain WHEN to use this tool vs others, WHAT prerequisites exist (must URL be pre-validated?), or HOW errors are handled.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 49 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 28 | - | v1 |
Placeholder 'hello' tool should not ship in production. It states 'More mortgage tools coming soon', this is a development artifact that adds noise and wastes agent reasoning.
No error handling documentation. How does the parser handle malformed PDFs, network failures, or invalid URLs? What errors can LLMs expect and how should they recover? e.g., 'URL must be HTTPS from whitelisted domain' is mentioned in description but no actual error response examples provided.
Parameter descriptions lack format constraints. 'pdf_url' is described as 'HTTPS URL to Loan Estimate PDF (must be from whitelisted domain)' but no regex pattern, example, or concrete validation rule is documented. LLMs may pass invalid URLs without knowing the constraint.