Loading ToolBench
Deterministic structured-data extraction from PDF documents
Three tools with clear naming (extract_fields, validate_against_schema, list_supported_document_types) and reasonable descriptions (100-150 chars each). All tools have input schemas with type definitions. However, parameter descriptions are minimal or missing context about constraints, formats, and error recovery. Output schemas are not documented. Error handling is basic (ok/error pattern) without recovery guidance. Missing tool annotations (readOnlyHint). Descriptions lack WHEN-to-use context and dependency hints.
Extract structured fields from an unstructured PDF using regex/heuristics. Args: pdf_path: Path to the PDF file. schema: JSON Schema describing the fields to extract (must have a non-empty 'properties' object). A built-in schema name (e.g. "invoice") or a path to a .json schema file is also accepted. Returns: {"ok": True, "data": {...}, "text_length": N} on success, or {"ok": False, "error": "..."} on failure (corrupt PDF, bad schema, ...).
List document types with pre-built schemas (e.g. invoice, resume, purchase_order).
Validate extracted data against a JSON Schema. Args: data: The data to check (e.g. the "data" from extract_fields). schema: JSON Schema to validate against. Returns: A structured report with passed/failed/missing field lists and a human-readable reason for each failure. Never raises: invalid schemas are reported via the "error" key instead.
Parameter descriptions lack format/constraint details. 'schema' param in extract_fields says 'JSON Schema describing the fields' but does not specify required structure (non-empty 'properties' object), accepted formats (dict/string/path), or error cases.
Output schemas not documented. Tools return dicts with 'ok', 'data', 'error' keys but LLM does not know the structure of 'data' field or what fields to expect in validation results. Agents cannot plan downstream calls without guessing.
Error responses lack recovery guidance. 'ok: False, error: ...' tells LLM an error occurred but not what to do next (retry? ask user? call a different tool?). No error categorization (retryable vs user-fixable vs fatal).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 57 | 2026-07-28+ | v2 |
Tool annotations missing. No readOnlyHint, destructiveHint, or idempotentHint declared. Clients cannot determine if tools are safe to retry or have side effects.
list_supported_document_types description (56 chars) is too brief. Does not explain when to call it (discovery before extract_fields?) or what structure it reveals.