Loading ToolBench
RFC analysis agent with empirical validation comparing custom API tools vs WebSearch-only implementation
Three tools with complete schemas and descriptions, but descriptions are generic and lack LLM-optimized guidance. All tools are read-only with proper input schemas (object types, required fields, descriptions). However, descriptions lack actionable context (when to use, what to do next, dependencies). Output schemas are not documented, LLMs cannot predict response structure. Error handling returns plain text without recovery guidance. No parameter constraints (enums, ranges) despite opportunities (e.g., limit should have min/max). Tool names follow verb_noun pattern correctly.
Fetch detailed information about a specific RFC by number or name
Fetch the full text content of an RFC
Search for RFCs by keyword or topic
Output schemas not documented. LLMs cannot predict response structure or plan downstream tool calls. search_rfcs returns a formatted text block; get_rfc returns markdown; get_rfc_text returns truncated text. No structured field definitions provided.
Descriptions lack LLM-optimized guidance. 'Search for RFCs by keyword or topic' (50 chars) does not explain WHEN to use this vs get_rfc, what query format is expected, or what the response contains. Missing dependency hints and use-case context.
No parameter constraints. 'limit' parameter in search_rfcs has no min/max bounds, LLM could pass 0, 1000000, or negative values. No enum for rfc_identifier format validation (accepts '7540', 'RFC7540', 'rfc7540' but this flexibility is undocumented).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 56 | 2026-07-28+ | v2 |
Error handling returns plain text without recovery guidance. 'Error searching RFCs: <exception>' or 'RFC not found' gives LLM no actionable next step. Should indicate: is this retryable? Should user provide different input? Are there alternatives?
get_rfc_text truncates at 50KB without explaining why or offering pagination. LLM cannot know if truncation occurred or how to fetch the rest. Should return structured response with total_length, truncated flag, and guidance on fetching full text.