Loading ToolBench
Two tools with reasonable naming (verb_noun pattern) but significant gaps in schema completeness and parameter documentation. get_all_device has a lengthy description (500+ chars) that exceeds best practice and lacks structured output schema. send_operate has a complex oneOf schema but missing descriptions for individual parameters within the schema objects. Neither tool documents error handling or recovery paths. No tool annotations (readOnlyHint/destructiveHint) present despite clear semantic differences (read vs write).
You will receive a JSON object containing information about smart home devices, groups, and scenes. The structure is as follows: ## Top-Level Fields - deviceAndDpInfoDTO: List of devices - deviceGroupAndDpInfoDTO: List of device groups - sceneInfoDTO: List of scenes ## Device Field Structure (deviceAndDpInfoDTO) Each device object includes: - deviceId (string): Device ID, corresponds to device_id in the control interface - deviceName (string): Device name - assetName (string): Room name - deviceDpInfoVOList (array): List of functional data points, each item includes: - key (string): Function key, corresponds to property in the control interface - name (string): Function name - value: Current value of the function point - specs (string): Value range or mapping description (in JSON string format) - type (string): Data type, can be int, bool, or string ## Group Field Structure (deviceGroupAndDpInfoDTO) Each group object includes: - id (string): Group ID, corresponds to group_id in the control interface - name (string): Group name - deviceDpInfoVOList: Same as above, representing the list of controllable function points for the group ## Scene Field Structure (sceneInfoDTO) Each scene object includes: - sceneId (string): Scene ID, corresponds to scene_id in the control interface - sceneName (string): Scene name Use this structure to understand the control logic for devices, groups, and scenes, and generate JSON control commands (e.g., for use with the send_operate tool).
You need to generate JSON data for control operations based on devices, groups, or scenes. Below is the data format and example instructions: ### Control Command Format (Array Form, each object represents a control instruction): Each object may contain one of the following identifiers: - device_id: Indicates device control (corresponds to a specific device) - group_id: Indicates group control (controls a collection of devices) - scene_id: Indicates scene control (triggers a predefined scene) Additional fields: - property: The functional key representing the control point of a device (e.g., switch, brightness, etc.) - value: The target value for the function point (can be a number, boolean, or string depending on the type) - ext_data: An object for extended control parameters, such as delayed execution; can be an empty object {}
get_all_device lacks output schema documentation. Tool description explains the response structure in prose, but no formal JSON Schema is provided for the return value. LLMs cannot reliably extract fields or plan downstream calls without a documented schema.
send_operate input schema uses oneOf with three object variants, but individual properties (device_id, group_id, scene_id, property, value, ext_data) lack descriptions. LLMs cannot infer parameter semantics from names alone; descriptions are mandatory.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear semantic differences. get_all_device is read-only; send_operate is destructive (modifies device state). Annotations guide agent planning and safety.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 44 | 2026-07-28+ | v2 |
Error handling is absent. Neither tool documents error codes, recovery paths, or actionable error messages. The send_operate description mentions error code ranges (1xx, 2xx, 3xx) but the tool does not document how errors are returned or what the LLM should do on failure.
get_all_device description is 500+ characters, exceeding the 10 - 1024 character best practice (p90=392). Verbose descriptions waste tokens and bury key details. Condense to ~150 chars and move examples to a prompt resource.