Loading ToolBench
A KYC (Know Your Customer) GraphRAG agent built with OpenAI Agent SDK. It features Neo4j MCP Server and Text2Cypher integration with Ollama for querying and analyzing customer financial data, accounts, and suspicious transaction patterns in a knowledge graph.
This KYC GraphRAG Agent server has 9 tools with generally adequate descriptions (10 tools have descriptions; 1 lacks description detail), but schemas are present and mostly complete. However, several critical gaps prevent a higher score: (1) parameter descriptions are sparse or missing in several tools, e.g., find_customer_rings has a customer_id parameter marked 'not implemented' but still defined; (2) output schemas are not explicitly documented, requiring inference from code; (3) error handling provides no recovery guidance or categorization; (4) no tool annotations (readOnlyHint/destructiveHint); (5) STDIO transport is a hard cap at 50. The server is functional for a specialized domain (KYC/AML) but lacks production-grade polish expected for general-purpose agent consumption. Naming is consistent (verb_resource pattern), and all tools follow appropriate semantic grouping for financial compliance workflows.
Create a Memory node and link it to specified customers, accounts, and transactions
Detects circular transaction patterns (up to 6 hops) involving high-risk customers. Finds account cycles where the accounts are owned by customers matching specified risk criteria (watchlisted and/or PEP status).
Find customers of interest (on watchlist or PEP) involved in account rings (cycles up to 6 hops).
Generate a Cypher query from natural language using a local finetuned text2cypher Ollama model
Get Customer details including its Accounts and some recent transactions. Limits the number of most recent transactions per account.
Given a customer_id, return all information in the Customer node and the name of all Accounts owned by this customer.
STDIO transport only, server is not remotely accessible and cannot be used by hosted MCP clients. This is a hard architectural limitation that prevents production deployment in cloud environments.
Output schemas are not explicitly documented in tool definitions. The code infers return structures from code logic (e.g., get_customer_info returns {customer, account_names}), but no formal schema is visible in the tool registration. LLMs cannot reliably plan downstream calls without seeing output schemas.
Missing tool annotations. No tools declare readOnlyHint (all are marked READ_ONLY or WRITE in metadata, but not in schema annotations), destructiveHint (create_memory writes state), or idempotentHint. Agents cannot safely retry without explicit guidance.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 56 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 39 | - | v1 |
Returns customer details if the customer is employed by more than 2 companies, otherwise returns None.
Returns True if the customer is involved in a suspicious ring (cycle up to 6 hops), otherwise False.
Check if a customer is linked to a "hot property" (address shared with more than 20 other customers).
find_customer_rings declares a customer_id parameter with description 'Specific customer to focus on (not implemented)', this is a parameter that does nothing. Either remove it or implement it. Leaving unimplemented parameters confuses agents.
No error categorization or recovery guidance. Tools return None or empty results on failure, with no indication whether the error is retryable, user-fixable, or fatal. Error responses lack actionable next steps (e.g., 'Customer not found. Try find_customers_in_rings() to search by criteria.').
Parameter descriptions are incomplete or missing context. For example, find_customers_in_rings has only one parameter (limit) visible in the tool registration, yet the docstring mentions 'is_customer_in_watchlist' and 'is_customer_pep' parameters that are not in the schema, these were likely removed but docstring not updated.
No pagination support or result limits in tool descriptions. find_customers_in_rings returns results up to 'limit' parameter (default 50), but description does not warn about result size or how to handle large datasets. Tools should declare max typical response size.
is_customer_linked_to_hot_property description is incomplete (source code cut off mid-query). Cannot fully assess parameter or output schema completeness.
No dry-run or confirmation pattern for destructive operations. create_memory writes to the database without any preview or confirmation mechanism. Agents could create erroneous memory nodes.
Neo4j credentials exposed via environment variables without server-side validation. Tools load NEO4J_URI, NEO4J_USER, and NEO4J_PASSWORD directly; credentials are not validated or rotated. If credentials are compromised, all database queries are compromised.