Loading ToolBench
Multi-tool AI agent with RAG, finance, web search, document processing, and company knowledge base capabilities. Built with LangGraph, supports long-term memory via PostgreSQL, checkpointing via Redis, and MCP integration.
The PA Agent server has 11 tools with highly variable quality. While 7 tools (inspect_file, summarise_file, extract_tables, ocr_image, save_uploaded_file, get_stock_quote, get_stock_news) have reasonable descriptions and visible schemas in the docs_tools.py and finance_tools.py files, several critical issues undermine the overall quality. Tool naming is mostly verb-driven and clear (inspect_, summarise_, extract_, ocr_, save_, get_), but parameter descriptions are inconsistent, some params lack descriptions entirely (e.g., image_path_or_url in ocr_image has a description, but the schema structure is not fully visible in source). The coinmarketcap_mcp tool is only referenced in app/mcp/servers.py with minimal description ('Access CoinMarketCap cryptocurrency data via MCP server') and no visible schema or implementation, scored at ceiling of inference penalty. Error handling is minimal, most tools return either a dict with an 'error' key or a string, but none include recovery guidance ('Try search_users() first') or error classification (retryable vs fatal). Output schemas are documented informally in docstrings but not formally declared. No tool annotations (readOnlyHint, destructiveHint, idempotentHint) are visible in the code. The server lacks pagination support (tools like get_stock_news have a max_items limit but no cursor or offset), and save_uploaded_file is a destructive operation that accepts an overwrite flag but provides no confirmation step or dry-run capability.
Access CoinMarketCap cryptocurrency data via MCP server.
Preview tabular data. • **CSV** – returns the first *head_rows* as JSON records. • **PDF** – extracts every table via *tabula‑py* (Java required). Any other extension → explanatory error message.
➜ Latest Yahoo‑Finance headlines for tickers (max `max_items`). Args: tickers: comma‑ or space‑separated string, or list of stock symbols. summarise: If True, add a 1‑sentence LLM summary per item. max_items: Number of headlines per ticker (up to 10). Returns: JSON string of either a single dict {"ticker": str, "news": [ {title, body, summary?}, … ] } or a list of such dicts for multiple tickers.
Fetch the latest quote and basic market data for one or more tickers. Args: tickers: comma- or space-separated string, or list of symbols. Returns: A list of dicts, each with keys: - ticker (str) - price (float) - currency (str) - previous_close (float) - open (float) - day_range (str)
Quick health‑check for *any* file. Returns path (local), byte‑size, guessed MIME type, and the first *head_chars* characters (or a <binary …> placeholder for non‑text blobs).
Three tools (web_fetch, tavily_search, lookup_company_policy) have descriptions visible in docstrings but NO input schemas visible in source code.
coinmarketcap_mcp is referenced only as an external MCP server integration in app/mcp/servers.py with no visible implementation, schema, or detailed description.
No tool includes error recovery guidance. Tools return errors (e.g., 'Cannot read file: {exc}' in summarise_file, 'Unsupported file type' in extract_tables) but do not advise the LLM what to do next (e.g., 'Try inspect_file() first to verify the file exists'). Violates pattern:recovery-guide.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 49 | <=2025-11-25 | v2 |
| 2026-03-09 | D | 51 | - | v1 |
Search the Knowledge Base. Use this to find info about: HR policies, IT setup, Engineering standards, Project Chimera, Security protocols, or Pricing. Args: query: The specific question or topic to search for.
Run Tesseract OCR on an image (PNG/JPEG/WEBP) and return the extracted text.
Persist a client‑uploaded file *exactly as received*. Parameters ---------- filename Destination path (absolute or relative). \ Intermediate directories are created automatically. content_b64 Raw base‑64 payload from the front‑end. overwrite If ``False`` and *filename* already exists → return an error. Returns ------- { "path": str, "size_bytes": int } on success, or { "error": str }.
LLM synopsis for small *textual* files (≤ ~15 kB recommended). Supported extensions -------------------- • .pdf – via ``PyPDFLoader`` (lazy import) • everything else is loaded as plain UTF‑8 Notes ----- 1. The document is **not** chunked into Pinecone – this is a one‑off call. 2. Large files are truncated after the first ≈3×1000‑char segments.
Search the web in real‑time via Tavily. Args: query: Your search query. max_results: How many hits to return (up to 5). Returns: A list of dicts, each with: - url (str) - title (str) - content (str, up to ~20 000 chars) - answer (str, Tavily's concise extracted answer)
Fetch and clean one or more web pages. Args: url: single URL or comma‑separated list of URLs (e.g. "gazzetta.it, example.com") max_pages: how many URLs to actually fetch (default 1) Returns: { "pages": [ { "source": <url>, "title": <page title if any>, "content":<cleaned text body> }, … ] }
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) are present. save_uploaded_file is destructive (writes to disk with optional overwrite) but has no destructiveHint annotation. get_stock_quote and tavily_search are read-only but lack readOnlyHint.
save_uploaded_file is a destructive operation (overwrites files) with no confirmation step or dry-run mode. An agent could accidentally overwrite critical files. Violates pattern:confirmation-request.
Pagination support is minimal or absent. get_stock_news has max_items (up to 10) but no cursor or offset for continuation. Tools returning lists (e.g., extract_tables returns list of dicts for PDF) lack pagination parameters. Violates pattern:paginated-result.
Output schemas are documented informally in docstrings (e.g., 'Returns { "path": str, "size_bytes": int }') but not formally declared as JSON Schema in tool definitions. Tools returning complex objects should declare schema explicitly so LLMs know what fields to expect.
Parameters accept multiple types (e.g., tickers in get_stock_quote is 'string or array') but descriptions do not clarify format or provide constraints. LLMs may pass invalid values like a dict or single ticker without quotes. Description should state: 'comma- or space-separated string (e.g., "AAPL,MSFT") or array of strings (e.g., ["AAPL", "MSFT"])'.
No tool validates inputs before processing. If an LLM passes a negative value for head_chars or head_rows, or a URL with invalid scheme, the tool fails silently or returns a generic error. Early validation with actionable error messages would enable LLM self-correction in one retry.