Loading ToolBench
Local MCP-style server for Robot Framework output analysis and failure review.
Server has 13 tools with mostly complete schemas and descriptions, but significant gaps in parameter documentation, error guidance, and composition clarity. Most tool descriptions are adequate (100-300 chars) but lack actionable context for LLM selection. Parameters are typed but many lack descriptions or constraints. No tool annotations (readOnlyHint/destructiveHint). Error handling returns HTTP status codes without recovery guidance. Tools like ingest_results and generate_failure_matrix perform multiple operations (parse + store, scan + generate + write), violating single-responsibility. Output schemas are partially documented via Pydantic models but not exposed in tool definitions.
Compare the current result file with a previous run.
Run a read-only SELECT against the failure analysis SQLite DB. Returns {columns, rows, count}. Results are capped at 200 rows; response includes truncated=true when the cap is hit. Schema: runs(id, results_dir, hash, ingested_at) scored_tests(id, run_id, area, suite, name, tags, message, failure_type, priority, severity, defect_ids, is_quarantined, fp, base_score, xml_path, api_endpoint, escalated, received_code, response_error) Args: results_dir: Same directory passed to ingest_results. sql: A read-only SELECT statement.
Generate bug report data for a specific failed test.
Scan a results directory and produce a scored, prioritised failure report. Args: results_dir: Directory whose subdirectories each contain robot/output.xml. area_filter: Optional folder-name substring filter, e.g. 'Smoke'. output_path: File path to write the report. Extension sets format (.html or .md). output_format: "html" (default, interactive with chart) or "markdown".
Return failed tests from the loaded result file. Messages are truncated to 200 chars. Use limit to control how many are returned (default 50). Response includes total count.
STDIO transport only, not remotely accessible. Cannot be used by hosted MCP clients or cloud-based agents. Limits deployment to local development environments.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). LLMs cannot distinguish safe read-only tools from destructive ones (ingest_results, generate_failure_matrix write files). Agents may incorrectly retry or compose tools.
Error responses return bare HTTP status codes (404, 400, 500) without actionable recovery guidance. E.g., 'FileNotFoundError' → HTTPException(404) tells LLM nothing about what to try next. Should include 'Try ingest_results() first' or 'Available directories: ...'
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 51 | 2026-07-28+ | v2 |
Fetch all tests in one failure group by its fingerprint. Args: results_dir: Same directory passed to get_triage_queue. failure_pattern: The failure_pattern string from the TriageItem. area_filter: Optional area filter (same value used for the queue).
Return the keyword execution trace for a specific test.
Return detailed information about a specific test.
Return a prioritised investigation queue of failing test groups. Args: results_dir: Directory whose subdirectories each contain robot/output.xml. area_filter: Optional folder-name substring filter, e.g. 'Smoke'. top_n: Maximum number of groups to return (default 15). exclude_known_defects: Skip groups where all tests already have defectid= tags. exclude_infra: Skip pure test-infrastructure failures.
Parse Robot Framework output.xml files in results_dir into a local SQLite DB. Run before get_triage_queue or generate_failure_matrix. Returns run_id and total_failures.
Load a Robot Framework result XML file for analysis.
Load a failing test from output.xml and return debug context for the robot-tests skill. Returns debug_category, product_file_hints, keyword_trace (FAIL path only), innermost_failure, failure_message, tags, priority, severity, defect_ids, suite_path. Args: xml_path: Path to the output.xml (use xml_path from get_triage_queue). test_name: Test case name (use representative_test from get_triage_queue).
Return failure groups. test_names capped at 10 per group; use get_group_tests for the full list.
Multiple tools combine unrelated operations: ingest_results (parse XML + write SQLite), generate_failure_matrix (scan dir + analyze + write file). Violates single-responsibility. Should split into separate tools so agents can compose independently.
Parameter descriptions missing or generic for several tools. E.g., get_triage_queue's 'area_filter' lacks format guidance ('folder-name substring filter' is vague). get_test_details 'test_name' has no hint about case sensitivity or exact-match requirement.