Loading ToolBench
Bugsafe has 5 tools with clear action verbs (scan_, capture_, render_, audit_, create_) and reasonable descriptions (100-250 chars). All tools have input schemas with type definitions. However, parameter descriptions lack specificity on constraints, formats, and error recovery guidance. Output schemas are documented in docstrings but not formally structured. Error handling is minimal, tools return generic error strings rather than actionable recovery guidance. No tool annotations (readOnlyHint/destructiveHint) despite clear risk profiles (capture_crash and create_bundle are WRITE operations). Parameter validation rules are not documented (e.g., timeout bounds, format enum for render_bundle). Missing dependency hints and chaining IDs in responses.
Verify that no secrets remain in a bundle after redaction. Args: bundle_path: Path to the .bugbundle file to audit. Returns: Dictionary with audit results: - passed: Boolean indicating if no secrets were found - leaks: List of pattern names that still matched (potential leaks)
Run a command and capture crash output with automatic secret redaction. Args: command: Shell command to execute (e.g., "python script.py"). timeout: Maximum execution time in seconds (default: 60). Returns: Dictionary with capture results including: - exit_code: Process exit code - stdout: Redacted standard output - stderr: Redacted standard error - duration_ms: Execution time in milliseconds - redaction_summary: Count of secrets redacted by category - timed_out: Whether the command timed out
Run a command and create a complete .bugbundle file. Args: command: Shell command to execute. output_path: Where to save the bundle (default: temp file). timeout: Maximum execution time in seconds (default: 300). Returns: Dictionary with: - bundle_path: Path to the created bundle - exit_code: Process exit code - redaction_summary: Count of secrets redacted - success: Whether bundle was created successfully
Render a .bugbundle file to human-readable format. Args: bundle_path: Path to the .bugbundle file. format: Output format - "markdown", "json", or "llm" (default: markdown). Returns: Rendered bundle content in the specified format.
Missing tool annotations for destructive operations. capture_crash (WRITE) and create_bundle (WRITE) lack destructiveHint annotations. Agents cannot distinguish safe read-only tools from state-modifying ones without explicit hints.
Parameter constraints undocumented. timeout parameter (capture_crash, create_bundle) lacks min/max bounds. format parameter (render_bundle) lacks enum constraint, description says 'markdown, json, or llm' but no formal enum. LLMs may pass invalid values.
Error handling lacks recovery guidance. render_bundle returns 'Error: Bundle not found at {path}', no suggestion to check path or list available bundles. audit_bundle returns bare dict on missing file, no actionable next step for the agent.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 56 | 2026-07-28+ | v2 |
Scan text for secrets and return redaction report. Args: text: Text content to scan for secrets. Returns: Dictionary with redaction results including: - found: Boolean indicating if secrets were found - total: Total number of secrets detected - categories: Count by secret type (API_KEY, TOKEN, etc.) - redacted_text: Text with secrets replaced by tokens
Output schemas lack chaining IDs. capture_crash and create_bundle do not return bundle_path or file references that downstream tools (render_bundle, audit_bundle) need. Agents must infer or guess file paths, forcing extra discovery calls.
Parameter descriptions lack format/range guidance. command parameter (capture_crash, create_bundle) does not specify shell syntax rules or escaping requirements. timeout defaults to 60/300 but no documented bounds (min 1, max 3600?). LLMs may pass invalid commands or extreme timeouts.