Loading ToolBench
A secure code modification assistant that analyzes repositories, selects relevant files, builds modification plans, and creates pull requests with safety guardrails including policy enforcement, diff validation, AST checks, and optional test verification.
SafeAgent has 8 tools with basic FastAPI registration but significant quality gaps. All tools have descriptions (positive), but most descriptions are under 100 chars and lack actionable context. Input schemas are present but minimal, parameters lack type constraints, enums, and detailed descriptions. The 'run' tool is particularly concerning: it performs irreversible operations (creates PRs) with no dry-run, confirmation, or error recovery guidance. No tool includes output schema documentation. Error handling is minimal (basic HTTPException). No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear risk levels. Tool naming is reasonable (analyze, run, list_sessions) but lacks verb clarity for some (run_manual is ambiguous). No pagination support on list_sessions despite database queries. Overall, definitions are functional but fall short of production-grade quality.
Analyzes a repository by cloning it and returning a list of files without loading their content.
Retrieves the unified diff for a specific SafeAgent execution session.
Retrieves a specific SafeAgent execution session by ID from the database.
Retrieves the execution trace for a specific SafeAgent session, including timing metrics and repair attempts.
Health check endpoint that returns the server status.
Retrieves a list of recent SafeAgent execution sessions from the database, ordered by creation time descending.
Executes the full SafeAgent pipeline: discovers files, selects relevant ones via LLM, loads their content, builds a modification plan, and executes it to create a pull request.
Irreversible 'run' tool lacks dry-run, confirmation, or rollback guidance. Creates PRs without agent confirmation, risking unintended repository modifications.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear risk levels. LLM cannot distinguish safe reads from destructive writes.
Parameter descriptions are minimal or missing. 'repo_url' and 'prompt' lack format constraints, length limits, or validation guidance. LLM cannot infer valid input.
No output schema documentation. Tools return JSON but LLM cannot predict field names, types, or structure for downstream chaining.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 59 | 2026-07-28+ | v2 |
Executes a manual test by modifying README.md with a deterministic change, generating a patch, and creating a pull request for validation purposes.
Error handling is generic (HTTPException with brief messages). No recovery guidance, categorization (retryable vs fatal), or actionable next steps for LLM.