Loading ToolBench
A Python library for building gRPC/ConnectRPC services with Pydantic models
pydantic-rpc is a gRPC/ConnectRPC framework library, not an MCP server. The evaluation treats the 30 example tools defined across various example files as if they were MCP tool definitions. Most tools have descriptions (good), but many lack sufficient schema detail, parameter descriptions are often generic, and there is significant duplication across examples (create_user, update_user, delete_user appear 4+ times with nearly identical signatures). The naming follows verb_noun convention (good), but the lack of distinct, purpose-driven tool names, and the fact that tools are framework examples rather than a cohesive, production MCP server, limits overall quality. Error handling is inconsistent: some tools document @error_handler usage, others do not. Output schemas are not explicitly documented for most tools. Many parameter descriptions are minimal (e.g., 'User name' with no constraints or examples of valid formats).
Ask the Olympics agent a question about Olympics locations
Ask the Olympics agent a streaming question about Olympics duration
Calculate with custom model serialization. The response will include a computed 'result' field.
Create a new book.
Create a new user.
Create a new user. This method uses @error_handler to customize validation error responses. If the request fails validation, the custom handler will be called with access to both the ValidationError and the raw request data.
Create a new user (async). This method uses @error_handler with Connect RPC error codes. Note: connect_code instead of status_code for Connect RPC!
Massive tool duplication across examples. create_user, update_user, and delete_user are defined 4+ times with nearly identical signatures. This violates the single responsibility principle and creates confusion about which tool to call. An LLM facing 30 tools with 12+ duplicates will waste reasoning cycles selecting between functionally identical options.
Parameter descriptions are minimal and generic. Examples: 'User name' (6 chars), 'User ID' (7 chars), 'User age' (8 chars). These fail to document constraints (e.g., age 0-120), valid formats, or why the parameter exists.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 55 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 0 | - | v1 |
Create a new user (synchronous). This method uses @error_handler to customize validation error responses.
Create a new user (synchronous). This method uses @error_handler with Connect RPC error codes. Note: connect_code instead of status_code for Connect RPC!
Delete a book by ID.
Delete a user (synchronous). No @error_handler - uses default behavior.
Delete a user.
Delete a user (synchronous). No @error_handler - uses default behavior.
Delete a user (async). No @error_handler - uses default behavior.
Delete a user. This method does NOT use @error_handler, so validation errors will be handled with the default behavior.
Get a book by ID.
Method with no request parameter (implicitly empty).
Get a user by ID.
Greet the user and identify the client if using mTLS.
Health check with empty request.
List all books with optional filtering.
Process user with custom field serialization. The name will be uppercased and age doubled due to serializers.
Say hello to the user
Update an existing book.
Update an existing user (synchronous). Uses @error_handler with default error handling.
Update an existing user (async). Uses @error_handler with default error handling.
Update an existing user.
Update an existing user (synchronous). Uses @error_handler with default error handling.
Update an existing user. This method uses @error_handler with default error handling (no custom handler function).
An operation that takes nothing and returns nothing.
Output schemas not documented. Tools like create_user, update_book, ask return responses (UserResponse, BookResponse, etc.) but the structure, field names, and types are not visible in the tool definition metadata. An LLM cannot infer downstream field names (e.g., does get_user return 'user_id' or 'id'?) without explicit schema documentation.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Tools are marked with a 'Risk' label (READ_ONLY, WRITE, DESTRUCTIVE) in the listing, but this metadata is not visible in the schema or description. LLMs need explicit destructiveHint on delete_user to understand the irreversible nature.
Inconsistent error handling documentation. Some tools mention @error_handler decorator (create_user, update_user, delete_user in error_handler examples), while others (say_hello, get_book, calculate) do not. This leaves the LLM uncertain about what happens when validation fails or the service errors out.
Field naming inconsistencies between tools. Some tools use 'id', others use 'user_id' or tool-specific IDs. E.g., create_user returns 'user_id', but get_user expects 'id'. This forces the LLM to reason about field mappings instead of chaining tools directly.