Loading ToolBench
MCP server for Microsoft LISA (Linux Integration Services Automation) framework. Exposes capabilities to discover and search test cases, filter tests by tier/priority/OS/platform, generate test suite and runbook YAML files, run tests via LISA CLI, parse and summarize test results, analyze failures with LLM (Azure OpenAI), and generate HTML/Markdown analysis reports.
The LISA MCP server presents a functional but inconsistently documented toolkit. Tool naming is mostly verb-first and actionable (discover_test_cases, generate_test_suite, run_tests), which is good. However, descriptions vary widely in quality, some are detailed (discover_test_cases has clear parameter breakdowns), while others are thin (check_lisa_installed has no substantive description). Parameter schemas are present for all inspected tools and use proper JSON Schema with types, but descriptions for input parameters are often minimal or missing context about expected formats, ranges, or constraints. Output schemas are largely undocumented, tools return JSON but callers don't have a declared schema of what fields to expect. Error handling is minimal; there's no guidance on recovery, retryability, or what the LLM should do if a call fails. Security considerations are present (api_key parameters are accepted, though this violates secret-injection pattern), and composition is reasonable (each tool has a single responsibility). The codebase is well-structured with modular tools modules, but the MCP registration layer (via FastMCP decorator) does not explicitly annotate tools with readOnlyHint/destructiveHint/idempotentHint, which modern agents rely on.
Add a test case to an existing runbook configuration.
Call an LLM to produce structured root-cause analysis for a single test failure.
Call an LLM to produce a run-level summary analysis of all test failures.
Construct a complete AnalysisReport combining test run metrics and LLM failure analyses.
Build a complete runbook for a specific test tier with platform and image configuration.
Verify that LISA is installed and accessible on the system.
Walk a LISA run directory and extract error context for all (or specified) tests.
STDIO transport only, not remotely accessible. Server cannot be used by hosted MCP clients or integrated into cloud-based agent platforms. This is a hard architectural limitation.
API credentials (api_key, endpoint) exposed as tool parameters in analyze_failure and analyze_run. This violates secret-injection pattern, credentials must be server-side injected via environment or vault. If these tools are called, credentials appear in agent traces and prompt history.
Output schemas not documented. Tools return JSON strings but callers have no formal declaration of field names, types, or structure. For example, discover_test_cases returns a complex nested structure, but there's no schema. This forces LLMs to parse and reason about undeclared fields.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 47 | 2026-07-28+ | v2 |
| 2026-03-09 | D | 58 | - | v1 |
Scan a local LISA repository and return all matching test cases.
Read the tail of a log file and extract lines near error signals.
Generate an HTML report from an AnalysisReport.
Generate a Markdown report from an AnalysisReport.
Generate a LISA runbook YAML configuration from structured test criteria.
Generate Python source code for a new LISA test suite.
Get the full metadata for a specific test case.
Return all unique functional areas (domains) in the LISA repository.
Parse a LISA JUnit XML results file and extract test outcomes.
Execute LISA tests via the lisa CLI command.
Save an AnalysisReport to disk as JSON, HTML, and/or Markdown files.
Free-text search across test case names AND descriptions in a LISA repo.
Generate a summary statistics object from parsed test results.
Validate a LISA runbook YAML file for syntax and structural correctness.
Write a runbook configuration to a YAML file.
Tool annotations (readOnlyHint, destructiveHint, idempotentHint) not present. Agents cannot automatically determine which tools are safe to retry or which modify state. run_tests, write_runbook, generate_test_suite_code, and others that mutate state should be marked destructiveHint=true or idempotentHint=true.
Minimal error handling guidance. Tools return JSON error messages (e.g., '{"error": "Test case not found"}') but do not guide the LLM on recovery. If discover_test_cases returns zero results, should the LLM try a different filter? Retry? Ask the user? No guidance.
Incomplete parameter descriptions. Many parameters lack examples of expected format or constraints. For instance, 'run_tests' has output_dir and timeout_seconds parameters with descriptions but no guidance on directory structure, timeout range (is 3600 reasonable?), or what happens if the timeout is exceeded.
Enum constraints missing. Parameters like 'tier' (T0 - T4), 'category' (functional|performance|stress|community), 'platform_type' (azure|hyperv), and 'formats' should use enum constraints, not free-form strings. This prevents hallucinated values.
Thin or missing descriptions for utility tools. check_lisa_installed has a one-line description with no guidance on when to call it (before what?) or what success/failure means. summarize lacks context on what it produces or why an agent would use it.
No confirmation or dry-run pattern for destructive operations. run_tests executes against real infrastructure. If an agent constructs the wrong runbook, tests execute wastefully. The tool should support a dry_run=true flag or require explicit confirmation.