A HackTheBox solver assistant that combines local RAG (Retrieval-Augmented Generation) with web search and MCP integration for command execution and machine exploitation guidance
Critical gaps across naming, descriptions, and schemas. Tool names lack consistent verb prefixes (e.g., 'is_mcp_available' vs 'check_mcp_availability'). Descriptions are present but generic (avg ~60 chars, well below 194-char baseline). Most critically, NO input schemas are visible in the provided source code, only parameter names and types are listed in the summary, not actual JSON Schema definitions. Two irreversible tools (execute_command, execute_command_mcp) lack confirmation/dry-run patterns. No output schemas documented. Error handling is absent from descriptions.
Execute a command safely with security validation
Execute a command using the MCP server
Extract the actual command from step text
Generate a report using the MCP server
Get the list of available tools from the MCP server
Get information about a machine from local documents
Get recommended commands for a specific machine and step
No input schemas visible in source code. Rubric requires JSON Schema with types and descriptions for every parameter. Only parameter names/types listed in summary, not actual schema definitions.
Irreversible tools (execute_command, execute_command_mcp) lack confirmation/dry-run patterns. Agents can execute arbitrary commands without safeguards.
Inconsistent naming conventions. 'is_mcp_available' uses 'is_' prefix (non-standard); should be 'check_mcp_availability' or 'get_mcp_status'. 'extract_command' lacks clarity on what 'extract' means vs 'parse_command'.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 32 | 2026-07-28+ | v2 |
Get information about a HackTheBox machine using RAG
Get a complete walkthrough for a specific machine from local documents
Get relevant content for a query from documents
Check if the MCP server is available
List all available machines in the data directory
Search for documents matching the query
Search for information about a HackTheBox machine
Update the document index
Descriptions are generic and under 100 chars (avg ~50 chars vs 194-char baseline). Missing WHEN to use, prerequisites, and return value guidance. E.g., 'Get information about a HackTheBox machine using RAG' does not explain when to call it vs get_local_machine_info.
No output schemas documented. LLMs cannot plan downstream tool calls or extract required fields (e.g., does list_available_machines return array of strings or objects with metadata?).