A LangGraph-based application that integrates multiple tools including web search, stock price lookup, calculator, PDF RAG, and MCP-based tools. Supports chat with persistent conversation history via SQLite checkpointing.
This MCP server exhibits significant quality gaps across naming, descriptions, parameters, and schemas. The server defines 8 tools but has 3 exact duplicates (calculator appears twice as tools #1 and #4; get_stock_price appears 3 times as tools #2, #5, #8; search appears twice as tools #3 and #7), indicating poor tool inventory management. Descriptions are present but generic and often include problematic example values (e.g., 'AAPL', 'TSLA' in get_stock_price). Parameter descriptions are minimal (1-3 words in many cases). Input schemas exist with types but lack critical constraints like enums for operation values in calculator or symbol validation in get_stock_price. No output schemas are documented. Error handling is minimal, tools return dicts with 'error' keys but provide no recovery guidance. The calculator tool accepts any operation string and returns an error, rather than constraining to enum values. get_stock_price exposes an API key in the source code (C9PE94QUEW9VWGFM), a critical security violation. rag_tool includes vague 'optional' thread_id parameter with no explanation of when it's required.
Perform a basic arithmetic operation on two numbers. Supported operations: add, sub, mul, div
Perform a basic arithmetic operation on two numbers. Supported operations: add, sub, mul, div
Fetch latest stock price for a given symbol (e.g. 'AAPL', 'TSLA') using Alpha Vantage with API key in the URL.
Fetch latest stock price for a given symbol (e.g. 'AAPL', 'TSLA') using Alpha Vantage with API key in the URL.
Fetch latest stock price for a given symbol (e.g. 'AAPL', 'TSLA') using Alpha Vantage with API key in the URL.
Retrieve relevant information from the uploaded PDF for this chat thread. Always include the thread_id when calling this tool.
Exact duplicate tool definitions. Calculator, get_stock_price, and search are registered multiple times (8 total tools, 4 unique). This confuses agent selection logic and wastes token budget on redundant definitions.
Hardcoded API key exposed in source code. get_stock_price contains plaintext Alpha Vantage key 'C9PE94QUEW9VWGFM' in langgraph_tool_backend.py line 50. This violates secret-injection pattern and compromises API security.
No output schemas documented. Tools return dicts but the structure is not formally specified. LLMs cannot predict which fields will exist or their types. For example, calculator returns {first_num, second_num, operation, result} or {error}, and get_stock_price returns r.json() with no structure definition.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 45 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 32 | - | v1 |
Web search using DuckDuckGo search engine
Web search using DuckDuckGo search engine
Parameter 'operation' in calculator accepts any string, then returns error if invalid. Should use enum constraint: ["add", "sub", "mul", "div"]. This forces LLMs to guess or suffer trial-and-error failures.
get_stock_price includes example stock symbols ('AAPL', 'TSLA') in the description. LLMs frequently reuse example values literally rather than adapting to actual user input, causing failed requests for unintended symbols.
rag_tool parameter 'thread_id' marked optional with vague description 'Thread ID for the chat session (optional)'. Description does not explain when thread_id is required vs. optional, or how it affects RAG results. Undocumented parameter relationships cause silent misuse.
search tool description is minimal (3 words). No guidance on query format, result count, result structure, or when to use this vs. get_stock_price for financial data. Baseline requires 10-1024 characters with clear WHAT/WHEN/WHY.
Error responses are generic. calculator returns {error: str(e)} and get_stock_price returns raw API JSON on failure. No recovery guidance: 'User not found. Try search_users() first.' Agents cannot determine if error is retryable or fatal.
calculator tool does not prevent division by zero gracefully at the schema level. Returns {error: 'Division by zero is not allowed'} at runtime. This should be caught and described in parameter constraints or tool logic before LLM invocation.