Automated ML deployment optimizer — finds the fastest feasible backend, quantization, and batch size for your model.
Two tools with complete schemas and reasonable descriptions, but significant gaps in parameter documentation and error handling. optimize_deployment has a well-structured schema with 6 parameters, all typed and with descriptions. list_supported_models is trivial (no params). However, parameter descriptions lack actionable constraints (e.g., no enum for 'objective' or 'gpu', no range for 'budget'). No output schema documented. No error handling guidance. Tool names follow verb_noun pattern correctly. Descriptions are adequate (100-150 chars) but could be more prescriptive about when to use each tool.
List the model names that deploy-agent can optimise.
Optimise deployment of a model by searching over backends, quantization, and batch sizes. Tries pytorch_eager, torch_compile, and onnxruntime with fp32/fp16/int8 quantization. Returns the best configuration that meets the latency and memory constraints.
Missing enum constraints on 'objective' and 'gpu' parameters. LLMs may hallucinate invalid values like 'minimize_memory' or 'tpu'. Should declare objective as enum: ['maximize_throughput', 'minimize_latency'] and gpu as enum: ['auto', 'cuda', 'cpu'].
No output schema documented. Tool returns JSON string but LLMs cannot infer structure of 'optimization report' or what fields to expect. Should document: {best_config: {backend, quantization, batch_size, ...}, trials: [...], metrics: {...}}.
No error handling guidance. If model_name is invalid or optimization fails, tool returns raw JSON error. Should return actionable errors like 'Model resnet99 not found. Call list_supported_models() to see available models.'
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 54 | 2026-07-28+ | v2 |
Parameter 'budget' lacks range constraints. Description says 'Number of trial configurations' but no min/max. LLMs may pass budget=1000000, causing timeout or resource exhaustion. Should specify: 'budget: integer, 1-1000 (default 25)'.
list_supported_models description is generic ('List the model names...'). Should explain when to call it: 'Call this first to discover available torchvision models before invoking optimize_deployment. Returns ~5 common models plus note that any torchvision name is accepted.'