Loading ToolBench
Multi-protocol server infrastructure for GraphRAG knowledge graph querying with gRPC, HTTP, WebSocket, and MCP support. Includes filesystem operations, SQLite database access, and GraphRAG entity resolution.
This is a gRPC-based server infrastructure with 7 tools, but the evaluation is severely constrained by incomplete source visibility. Only the Dockerfile, package.json files, and a non-functional code snippet (grpc-server/server.js appears truncated) are provided. NO actual tool definitions, schemas, or descriptions are visible in the source code. The assessment below is based on the tool metadata provided in the REPO header only. All per-tool scores are capped at 50 per the hard rules (cannot infer tool existence from metadata alone), but most are scored lower due to missing schema and description evidence.
Incremental graph building with bidirectional streaming
Stream context retrieval for GraphRAG with entity information
Service health monitoring endpoint
Query the knowledge graph with GraphRAG
High-performance entity resolution with similarity matching
Real-time graph updates through server streaming
Stream graph traversal results through server streaming
Source code truncated or missing: grpc-server/server.js is incomplete. Cannot verify actual tool definitions, input/output schemas, parameter validation, or error handling logic. All assessment is inference from metadata only.
No parameter descriptions visible in source. Metadata shows parameter names and types (e.g., 'query' string, 'graph_id' string, 'model' string for QueryGraph), but NO descriptions explaining what each parameter controls, what formats are valid, or what constraints apply. However, since schemas ARE visible in metadata, scored conservatively at 15-25 instead.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 36 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 41 | - | v1 |
Tool descriptions are generic and under 100 characters. QueryGraph: 'Query the knowledge graph with GraphRAG' does not explain when to call it, what the output contains, or how it differs from GetContextStream or TraverseGraph. Similar issues across all tools.
Output schemas are not documented in the provided metadata. Baseline requirement: '100% of A+ tools have documented return types.' Tools return 'stream' or complex results (e.g., GetContextStream returns 'context retrieval'), but no schema shows what fields the LLM should expect, how pagination works, or what metadata is included.
Tool names lack clear action verbs. 'QueryGraph' is acceptable, but 'TraverseGraph' and 'GetContextStream' are vague, LLMs may confuse these three graph-reading tools. Per pattern: 'When multiple tools operate on the same resource, their names must make the distinction obvious.' Consider: query_graph, traverse_graph_by_path, retrieve_entity_context.
'BuildGraph' accepts a 'document' object parameter but no schema or description shows what fields the document must have (e.g., title, content, metadata, format).
Error handling and recovery guidance are not visible in metadata or source. Per pattern:recovery-guide, 'Error responses must tell the LLM what to do next.' No evidence of categorized errors (retryable, user-fixable, fatal), actionable error messages, or recovery instructions.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) are visible. Metadata shows 'Risk: READ_ONLY' or 'Risk: WRITE' for each tool, but these are not standard MCP tool annotations. Per protocol, current spec requires explicit annotation capability for agent decision-making.
Parameter 'model' in QueryGraph defaults to 'gpt-5-nano', a non-existent model name (as of 2025). This is either a typo or test placeholder. Default values should reflect production-grade models and must not cause silent failures.
Pagination not documented. Tools like GetContextStream and StreamGraphUpdates accept 'max_context_size' and 'max_results' limits, but no description shows what the response structure is, whether results are paginated, how to fetch the next page, or what the total count is. Per pattern:paginated-result, tools returning lists must support pagination.