Loading ToolBench
Model Context Protocol server that exposes TP-Link Omada controller APIs
This server has 24 tools with consistent naming (all verb_noun), present descriptions, and explicit JSON Schema input types. However, there are critical gaps: (1) Output schemas are NOT documented, LLMs cannot plan downstream calls or extract fields. (2) Descriptions are generic and lack actionable context (10-30 chars is below the 194-char baseline). (3) No error handling guidance is visible, agents cannot recover from failures. (4) No parameter constraints (enums, ranges) for critical fields like 'type' in getPortForwardingStatus. (5) Tool chaining will suffer because responses don't explicitly document what IDs/references are returned. Example: listSites returns OmadaSiteSummary[], but the schema is never shown; downstream tools like listDevices(siteId) depend on getting a valid siteId from that response, yet the field name and type are invisible to the LLM. Similar pattern for all 24 tools, LLMs must infer response structure.
Get all users (both cloud and local) in a grid/app view format.
Get application control categories (families) for a site, listing available app category definitions used in application control rules.
Get application control rules for a site, listing application-level firewall rules that allow or block specific apps or app categories.
[DEPRECATED] Get MAC group profiles for a site. This is an alias for getGroupPolicyDetail with groupType="2" — use getGroupPolicyDetail instead.
[DEPRECATED] Get application control rules for a site. This is an alias for getAppControlRules — use getAppControlRules instead.
Get application control (DPI) status and configuration for a site.
Output schemas are not documented anywhere. LLMs cannot plan downstream tool calls or extract required fields from responses. Example: listSites() returns OmadaSiteSummary[], but the schema (fields, types, required) is invisible. Downstream tools like listDevices(siteId) depend on getting a valid siteId from that response, yet the field name and structure are never shown to the LLM.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 14 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 35 | - | v1 |
Get detailed configuration for a specific client-to-site VPN server by ID, including protocol, port, authentication type, and allowed client settings.
Get the Google LDAP profile configuration for a site, including domain, binding credentials, and user/group base DN settings.
Get the mail server connection status for the controller.
Get the global RADIUS server configuration for the controller.
Get the SSL VPN server configuration, including port, protocol, and authentication settings.
List all client-to-site VPN server configurations on the site, including server type, IP, and authentication settings.
Tool descriptions are generic and under the 194-character baseline. Examples: 'List all sites available in the Omada controller' (52 chars), 'List all devices in a site' (26 chars), 'Get a specific device by MAC address or device ID' (50 chars). Descriptions lack actionable context: When should I call this? What are the prerequisites? What does it return? What should I do with the result? LLMs cannot distinguish between similar read-only tools without richer descriptions.
No error handling guidance visible. When a tool fails (e.g., invalid siteId, device not found, rate limit hit), LLMs cannot determine if the failure is retryable, requires user input, or is fatal. No recovery paths documented. Example: if listDevices returns an error for an invalid siteId, the LLM cannot self-correct without explicit guidance.
Critical parameters lack constraints (enums, ranges, patterns). Example: getPortForwardingStatus accepts type: string with description 'User' or 'UPnP', but no enum constraint. The LLM may pass 'user' (lowercase), 'upnp', 'UPnP', or invent values. Similar issue: listDevicesStats filterDeviceSeriesType is unconstrained; listClientsActivity start/end are numeric with no min/max bounds (could be negative, too large, causing API errors or timeouts).
No tool chaining documentation. When a user asks 'Show me the rate limit profiles and apply one to client MAC X', the agent must figure out: (1) call getRateLimitProfiles() to get profile IDs, (2) extract a profileId from the result, (3) call setClientRateLimitProfile(clientMac, profileId). The response schema for getRateLimitProfiles is invisible, the agent cannot know if the field is 'profileId', 'id', 'profile_id', or 'profileID'. This forces the LLM to guess or make extra discovery calls.
Parameter descriptions are minimal or missing context. Example: 'Optional search by MAC addresses' (listDevicesStats.searchMacs) does not explain: Format (comma-separated? array notation?)? Single MAC or multiple? Case-sensitive? Wildcards allowed? Similar: 'Optional filter by tag' (listDevicesStats.filterTag), what is a 'tag' in this context? How are tags defined? Must the agent know tag names in advance or discover them first?
No pagination guidance for list tools. listDevices, listClients, getThreatList, listDevicesStats, and others accept pagination (page, pageSize). What is the maximum pageSize? What is the default? When there are no more results, what value indicates end-of-list (does nextPage=null, or page number stays the same)? Without this, agents may: (1) request invalid page sizes, (2) infinite-loop trying to fetch beyond the end, (3) miss results due to off-by-one errors.
Destructive tools (setClientRateLimit, setClientRateLimitProfile, disableClientRateLimit) lack a confirmation/dry-run pattern. An agent with a mistake could: (1) apply rate limits to the wrong client (if MAC address is mistyped), (2) disable rate limits unintentionally (blocking agent recovery). No guidance on whether these operations are reversible or how to undo them.
Parameter 'identifier' in getDevice and getClient is overloaded (MAC address OR device/client ID). Description says 'Device MAC address or device ID' but does not specify: (1) How the tool disambiguates between them (is MAC address always colon-separated? Is ID always numeric?). (2) What happens if both formats could match (e.g., '00:11:22:33:44:55', is this always MAC, or could it be an ID?). LLM may pass ambiguous values causing silent failures.
siteId parameter marked 'Optional' with default from config, but if no config default is set, what happens? Does the tool error with 'siteId required'? Or does it fail silently on the default site? Ambiguity here causes agents to: (1) fail without clear error, (2) apply actions to the wrong site. Every optional parameter with a server-side default must document: What is the fallback? What error if fallback is missing?