Loading ToolBench
API for managing Arduino projects, libraries, and board cores with arduino-cli (cached file listing)
The server presents 19 tools for Arduino project management with complete input schemas and descriptions. However, several critical gaps prevent a higher score: (1) tool descriptions are generic and lack LLM-specific guidance on WHEN to use each tool and what it returns; (2) output schemas are completely undocumented, callers cannot see what fields to expect, breaking the tool chain pattern; (3) parameter descriptions are minimal (1-3 words each) and lack format/constraint guidance; (4) no error handling patterns, tools will throw HTTPException but do not provide recovery guidance; (5) security is present (path validation, safe_name checks) but not documented in tool descriptions; (6) no pagination or result limits documented despite tools like list_projects returning file lists that could be large. Naming is generally good (verb_noun pattern) but descriptions fail the 50-200 character LLM-optimized guideline, averaging ~60 chars, which is too brief to guide agent selection or explain when to use vs similar tools (e.g. list_projects vs get_project_files). This server lands in the lower-C range, competent mechanics but poor agent-readiness.
Compile an Arduino sketch for a specific board and FQBN
Copy a library example project into a new Arduino project
Create a new Arduino project with optional initial sketch content
Delete an entire Arduino project directory
Get the list of files in a specific library
Get the list of files in a specific project
Install an Arduino board core by name
Install an Arduino library by name
Output schemas completely undocumented. Tools return data but callers cannot see what fields to expect (e.g., list_projects returns a list of project objects, but no schema shows whether each entry has {name: string, path: string, files: [string]} or something else). This breaks the tool-chain pattern and forces LLMs to guess at response structure.
Tool descriptions are 20-60 characters, far below the LLM-optimized 50-200 character baseline. E.g., 'Compile an Arduino sketch for a specific board and FQBN' (58 chars) does not explain WHEN to call compile_sketch vs other tools, what happens on success/failure, or whether it requires prior project setup. LLMs cannot select tools effectively with such minimal guidance.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 55 | <=2025-11-25 | v2 |
List all installed Arduino board cores
List all installed Arduino libraries with their file paths
List all cached Arduino projects with their file paths
Read the content of a file within a library
Read the content of a file within a project
Search for Arduino board cores by keyword using arduino-cli
Search for Arduino libraries by keyword using arduino-cli
Uninstall an Arduino board core by name
Uninstall an Arduino library by name
Upload a compiled sketch to an Arduino board via specified port
Create or update a file in a project
Parameter descriptions are minimal (1-3 words). E.g., 'Name of the Arduino project' for project_name does not specify format (allowed characters, length), or whether it can be auto-created if missing. LLMs cannot validate their own input and will pass invalid values (spaces, special chars) that the server rejects.
No error guidance in tool descriptions. Destructive tools (delete_project, uninstall_library, uninstall_core) do not warn that operations are irreversible or suggest confirmation patterns. Error responses (HTTP 400, 404, 500) are not documented, so LLMs cannot plan recovery. E.g., if delete_project fails with a 404, what should the LLM do? Retry? Call list_projects first?
No pagination or result-limiting documented. list_projects, list_libraries, list_cores, get_project_files, get_library_files all return unbounded lists from file system walks. If a user has 1000 projects, list_projects() returns 1000 entries, exhausting the context window. No limit or offset parameters are visible.
Security logic (path validation, safe_name checks) is implemented server-side but NOT documented in tool descriptions. LLMs do not know that project_name must match SAFE_NAME_RE (alphanumeric, dot, space, dash only) or that file_path cannot escape the project directory. This invites invalid calls and 400 errors with no recovery guidance.
Inconsistent parameter naming. write_project_file has 'sketch_content' (domain-specific) but read_project_file does not return 'sketch_content', the response schema is unknown. create_project also uses 'sketch_content'. This mismatch forces LLMs to infer field names from usage patterns rather than explicit documentation.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) visible in source. Risk field is present in the evaluation data but NOT in the actual tool registration in server.py. This means clients cannot distinguish safe vs dangerous tools without manual inspection.