Loading ToolBench
Multi-provider LLM chat with RAG knowledge base and MCP tool integration. Includes FastAPI backend, React frontend, and CLI. Supports Anthropic, OpenAI, Google Gemini, and Ollama models.
This MCP server demonstrates moderate quality with mixed strengths and significant weaknesses. The 13 tools are properly registered with names and schemas, but descriptions lack depth and many parameters are under-documented. Tool naming follows verb_noun convention (get_*, search_*) which is appropriate. However, descriptions are often vague or generic, parameter descriptions are minimal or missing, and output schemas are not documented. Error handling is minimal, most tools appear to return raw data without guidance on failure modes. The `_tool` decorator provides schema generation, but it is overly simplistic: it infers all parameters as strings by default unless type-annotated, and parameter descriptions come from docstrings rather than explicit schema metadata. Financial and weather tools are READ_ONLY, which is correct, but none declare this via toolAnnotations (readOnlyHint). Several tools (finance tools especially) likely return unfiltered API responses without result limiting or pagination guidance.
Evaluate a mathematical expression. Supports standard math functions (sqrt, sin, cos, log, etc.), abs, round, min, and max.
Compare multiple stocks side-by-side.
Get balance sheet data (assets, liabilities, equity, debt) for a ticker symbol.
Get cash flow statement (operating, investing, financing) for a ticker symbol.
Get company overview — name, sector, industry, description, and more.
Get the current date and time.
Get historical price data (OHLCV) for a ticker symbol.
Parameter descriptions are missing or minimal. The _tool decorator extracts docstring text for the tool description but does not extract per-parameter documentation. Example: get_stock_price accepts 'symbol' with description 'Stock ticker symbol', adequate but minimal. Many parameters (e.g. 'period' in get_income_statement) lack detail about valid values, constraints, or examples.
No enum constraints for parameters with discrete valid values. Example: get_income_statement and get_balance_sheet accept 'period' as a string ('annual' or 'quarterly'), but this is documented only in the parameter description, not as a JSON Schema enum. LLMs may hallucinate invalid values like 'monthly' or 'semi-annual'.
Inferred effective spec: 2025-06-18+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 56 | 2025-06-18+ | v2 |
| 2026-03-09 | F | 42 | - | v1 |
Get income statement data (revenue, net income, EPS) for a ticker symbol.
Get key financial ratios for a ticker symbol.
Get current stock price, change, and market cap for a ticker symbol (e.g. AAPL, GOOG, TSLA).
Get user's information about: - city - timezone - local_time Use this to personalize responses or determine the user's timezone.
Get current weather conditions for a city (e.g. New York, London, Tokyo). Returns temperature, humidity, wind speed, and conditions.
Search the knowledge base for information relevant to a query.
Output schemas are not documented. None of the tools include documentation about what fields they return, their types, or structure. Example: get_stock_price returns data from yfinance, are the fields 'currentPrice', 'currency', 'change', 'marketCap'? This forces LLMs to infer output structure, increasing errors and requiring extra discovery calls.
No error handling guidance. Tools do not document what happens on failure (invalid ticker, rate limit, network error, unavailable service). The mcp_tool wrapper in tools.py includes some error handling (checking for 'unavailable' and 'error' keys), but there is no guidance to the LLM about what to do if a request fails or how to recover.
Tool annotations missing. None of the tools declare readOnlyHint, destructiveHint, or idempotentHint. While most tools are read-only (appropriate for a RAG + data lookup system), this should be explicitly stated in the schema so agents and clients can reason about safety.
Vague or incomplete tool descriptions. Example: get_user_context description says 'Get user's information about: - city - timezone - local_time' but does not explain when to call it or how it personalizes responses. get_company_info says 'Get company overview, name, sector, industry, description, and more', 'and more' is vague. Descriptions should answer: What does it do? When should the LLM call it? What does it return?
No pagination or result limiting documented. Financial tools (get_historical_prices, etc.) likely return large datasets. There is no documented limit, pagination mechanism (page, offset, limit), or guidance on result size. This risks context window exhaustion and token waste.
Parameter type inference is simplistic. The _tool decorator defaults all parameters to string type unless explicitly type-annotated. Example: 'period' in get_income_statement is inferred as string, but the schema should clarify it is an enum ('annual'|'quarterly'). Type annotations are present in the function signature but the schema generation does not leverage them fully.