Loading ToolBench
AI Ops - Advanced artificial intelligence capabilities for Nautobot through multi-provider LLM architecture and Model Context Protocol
Scoring was not performed
Tool naming violates verb_noun convention. 'mcp_nautobot_openapi_api_request_schema' uses a noun ('schema') instead of an action verb like 'search_' or 'get_'. This makes it unclear to agents when to invoke this tool. The name should be 'search_nautobot_api_endpoints' or 'get_api_endpoint_schema'.
Output schemas are completely undocumented. The source code does not show what fields 'mcp_nautobot_openapi_api_request_schema' returns, only that it returns 'endpoint details including path, method, parameters, and response formats'. Without a documented schema, agents cannot plan downstream tool calls or extract the right data fields.
Generic API executor tool 'mcp_nautobot_dynamic_api_request' delegates HTTP semantics to the agent. This violates the single-responsibility principle, agents should not reason about REST verbs, query params vs body fields, and endpoint paths. This increases error rates and requires agents to parse verbose API docs. Wrap common operations (get device, create interface, update config) in domain-specific tools instead.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 27 | <=2025-11-25 | v2 |
| 2026-03-09 | D | 56 | - | v1 |
Parameter descriptions lack constraint details. 'method' is described only as 'HTTP method (GET, POST, PUT, PATCH, DELETE)', no guidance on when each applies. 'path' lacks format rules. 'params' and 'body' descriptions are generic. These descriptions fall short.
Error handling is not documented. The tool descriptions do not explain how to recover from 404 (endpoint not found), 401 (auth failed), or 422 (invalid request). Per pattern:recovery-guide, error responses must tell the LLM what to do next. Current descriptions provide no guidance.
Refresh tool naming is ambiguous. 'mcp_refresh_endpoint_index', what is the 'index'? Is it a database index, a search index, or an in-memory cache? The description says 'manually refresh the OpenAPI endpoint index', but 'index' is jargon. Rename to 'refresh_api_schema' or 'sync_nautobot_endpoints' for clarity.
No pagination support documented. If the search tool returns many endpoints, there is no evidence of limit, offset, or cursor parameters. The 'n_results' parameter on search is good, but no documentation of what happens if there are more results than requested.