Loading ToolBench
The Lambda MCP server provides 8 tools for GPU instance and filesystem management with reasonable naming conventions and clear problem domains. Tool names follow verb_noun pattern (list_*, start_*, stop_*, create_*, delete_*), which is consistent with baseline conventions. However, descriptions are inconsistent in quality and depth, some tools have good actionable descriptions while others are minimal. Parameter documentation is uneven: some parameters have clear type and descriptions, but the rubric requires that ALL parameters be explicitly documented with types and constraints. Most tools show proper input schemas with type definitions (gpu: string, region: string, etc.), but descriptions lack specificity around constraints, valid values, and error conditions. Output schemas are not documented, the server returns results but LLMs cannot predict response structure. Error handling guidance is minimal. The security posture is adequate (no credentials in parameters, API key managed via environment), but no permission gating or audit trail patterns are evident.
Check if a specific GPU type is currently available. Returns list of regions with availability.
Create a new persistent filesystem (storage volume) in the specified region
Delete a persistent filesystem (storage volume). The filesystem must not be in use by any running instances.
List all persistent filesystems (storage volumes) available in your account with usage information
List all available GPU instance types with pricing, specs, and current availability
List all currently running GPU instances with their status and connection details
Output schemas are not documented. Tools return results (instance details, filesystem info, pricing, etc.) but LLMs cannot predict response structure, field names, or types. This forces agents to infer output shape at runtime and makes tool chaining fragile.
Parameter descriptions lack constraint documentation. For example, 'gpu' parameter states 'GPU instance type (e.g., gpu_1x_h100, gpu_1x_a100)' but does not specify: are these the only valid values? Is there a canonical list? Should the agent call list_gpu_types first? Parameter 'region' similarly lacks guidance on format, valid values, or auto-selection behavior.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 63 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 0 | - | v1 |
Launch a new GPU instance. Returns instance ID and connection details. Optionally attach a filesystem (must be in the same region). If notification env vars are configured, will auto-notify when instance is SSH-able.
Terminate a running GPU instance
No error recovery guidance. Descriptions do not explain what the LLM should do if a call fails. For example, start_instance may fail if the GPU is unavailable or region is invalid, but the agent receives no guidance on recovery steps (e.g., call check_availability, try another region, call list_gpu_types).
Destructive operations lack confirmation/dry-run pattern. stop_instance and delete_filesystem are irreversible and destructive, but tool descriptions do not mention confirmation steps or dry-run capability. Agents have no built-in guard against accidental termination.
No pagination support. list_gpu_types, list_running_instances, and list_filesystems may return large result sets but tool definitions do not include limit, offset, or pagination parameters. Large unfiltered lists blow context window and degrade LLM reasoning.
Parameter 'name' for start_instance is optional and described only as 'Optional name for the instance'. No guidance on format, length limits, character restrictions, or naming rules. Agents may pass invalid values without error guidance.
No dependency hints in tool descriptions. start_instance accepts a 'gpu' parameter but does not guide the agent to call list_gpu_types or check_availability first. Agents may pass invalid GPU types without context.
Permission gating and audit trail patterns are absent. No indication that tools verify user authorization, log who called what, or support least-privilege agent configurations.