Loading ToolBench
Multi-engine LLM benchmark & monitoring CLI for Apple Silicon with MCP server support for inference monitoring and benchmarking
The server provides 5 well-named tools with clear, action-oriented verbs (check_*, get_*, list_*, detect_*, run_*). Descriptions are substantive (range 130-570 chars, well above the 34-char p10 baseline) and include WHEN to use each tool and what to expect. All tools have proper type annotations via @mcp.tool decorators. Tool annotations (readOnlyHint, openWorldHint, title) are present on most tools, supporting spec alignment. However, input schemas lack full JSON Schema formalization in the visible code, parameters are documented in descriptions but not exposed as structured type/property objects. The run_benchmark tool stands out with detailed parameter guidance (model, engines, compare, prompts, runs, context_size, card) but still lacks formal schema constraints like enum declarations and min/max bounds. Error handling is minimal, no recovery guides or retryable vs. fatal classifications.
Quick health check of all local LLM inference engines. Returns status ("ok", "degraded", "error"), which engines are reachable, memory pressure, thermal state, and GPU utilization. Use this FIRST when you need to know if inference is available. Responds in <500ms.
Auto-detect running LLM inference engines on this Mac. Uses a 3-layer detection cascade: 1. Config — engines remembered from previous runs (non-standard ports) 2. Port scan — default ports + extended range (8000-8009, 8080-8082) 3. Process detection — ps/lsof to find engines on any port. Discovered engines are auto-persisted to ~/.config/asiai/engines.json.
Get complete system + inference state: CPU load, memory, thermal, GPU metrics, all engines with their loaded models, VRAM usage, TCP connections, and inference activity. Use this when you need detailed information about the current state of the machine and all inference engines. More comprehensive than check_inference_health but takes slightly longer (~1s).
List all models currently loaded across all inference engines. Returns each model with its engine, name, VRAM usage, format, quantization, and context length. Use this to know what models are available for benchmarking or inference.
Run a performance benchmark on local LLM inference engines. WARNING: This is an ACTIVE operation that takes 30-120 seconds. It sends prompts to inference engines and measures tok/s, TTFT, and throughput. Only use when the user explicitly asks for a benchmark. Two modes: Engine comparison (default): benchmark one model across engines. Use model + engines parameters. Cross-model comparison: benchmark different models (optionally on different engines). Use compare parameter with model@engine syntax.
Input schemas not formalized as JSON Schema objects. Parameters are documented in prose descriptions but lack type/properties/required declarations, making it impossible for clients to validate or generate UI forms. The rubric requires 'Every parameter needs a description' AND type definitions for 80+ scores.
run_benchmark parameters lack enum constraints and numeric bounds. 'prompts' should declare valid enum values (code, tool_call, reasoning, long_gen). 'runs' parameter should declare min=1, max=10 in the schema, not just in prose. 'context_size' lacks a regex pattern or format.
No error handling or recovery guidance. Tools return dicts but do not document what happens on engine unavailability, network timeouts, or invalid inputs. The rubric requires 'Error responses must tell the LLM what to do next' and 'Categorize errors as retryable, user-fixable, or fatal.'
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 62 | <=2025-11-25 | v2 |
run_benchmark is marked Risk=WRITE but has no confirmation or dry-run mode. The rubric warns: 'Irreversible operations should support a dry-run or confirmation step.' A 30-120 second benchmark that may consume significant GPU/CPU resources should offer a confirm step to prevent accidental execution.
Output schemas not formally documented. Tools return dicts (e.g., check_inference_health returns {status, ts, engines, memory_pressure, memory_used_pct, thermal_level, thermal_speed_limit, gpu_utilization_pct}) but the expected structure is not declared in schema. Clients cannot validate responses or generate parsing logic.
detect_engines 'urls' parameter description includes an example value ('http://localhost:11434'). LLMs tend to reuse example values literally rather than adapting to the actual context.' Replace with format guidance or a regex pattern.