Loading ToolBench
Plugin-based FRC documentation agent as an MCP server for WPILib and vendor libraries
The WPILib MCP server has reasonably well-structured tools with clear purposes and documented schemas, but falls short of production-grade quality due to missing output schema documentation, generic descriptions, incomplete parameter documentation, and lack of error handling guidance. Four tools are defined with mostly coherent naming and adequate input schemas. Tool descriptions explain purpose and use cases adequately (140-180 chars), meeting the 10-1024 baseline, but lack actionable recovery guidance and dependency hints. Input schemas are complete with types and property descriptions for most parameters, but output structures are entirely undocumented, a critical gap. Parameter descriptions are present but sometimes generic (e.g., 'Programming language filter' without context). The server reads project files safely (READ_ONLY risk) but provides no error classification, recovery guidance, or validation feedback to guide LLM recovery.
Detect the FRC project's programming language and vendor libraries by scanning source files, build configs, and vendordeps. Useful for understanding what hardware/libraries the project uses.
Fetch the full content of an FRC documentation page. Automatically routes to the correct plugin based on URL domain. Returns cleaned text content suitable for answering questions.
List available FRC documentation sections/categories. Useful for browsing what documentation is available from each vendor.
Search FRC documentation across WPILib and vendor libraries (REV, CTRE, Redux, etc.). Returns ranked results with titles, URLs, and content previews. Use auto_detect=true to automatically filter by the project's detected language and vendors.
Output schemas completely undocumented. No return type definitions visible for any of the 4 tools. SearchResult, PageContent, DocSection types are referenced in code but never formally documented in tool output specifications. LLMs cannot plan downstream tool calls or extract relevant fields without documented return structures.
No error handling guidance or recovery instructions. Tools lack descriptions of failure modes, retryability, or what the LLM should do if a query returns no results, a URL is invalid, or project detection fails. Bare errors without context force LLM to guess recovery strategy.
Missing dependency hints and parameter relationships. 'auto_detect' parameter in search_frc_docs is confusing, description says 'Auto-detected if auto_detect=true' and mentions ignoring other parameters, but LLMs may not understand the mutual exclusivity. Parameter descriptions lack: 'If auto_detect=false, vendors and language are used; if true, they are ignored based on detected project context.'
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 66 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 46 | - | v1 |
Pagination and result limiting not fully addressed. search_frc_docs accepts max_results (1-25, default 10) which is good, but no documentation of what happens if results exceed max_results, are they ranked? Truncated? No 'next_cursor' or 'total_count' in documented output. list_frc_doc_sections provides no limits or pagination guidance.
No tool annotations for safety. Tools lack readOnlyHint, destructiveHint, or idempotentHint metadata, even though these are safe read-only operations that could declare that explicitly to guide agent behavior.
Parameter descriptions lack actionable format/constraint hints. 'vendors' parameter says 'specific like [wpilib, rev, ctre]' but does not define valid vendor names. 'version' defaults to '2025' with example '2024' but no description of version format or supported versions. LLMs must guess valid inputs.
Tool descriptions could be more specific about use cases and context. detect_project_context says 'Useful for understanding what hardware/libraries the project uses' but does not explain when to call it relative to search_frc_docs, or that auto_detect in search_frc_docs implicitly calls this. list_frc_doc_sections says 'Useful for browsing' without explaining when browsing is better than direct search.