Loading ToolBench
ASTRA defines 4 tools with basic schemas and descriptions. All tools have names starting with action verbs (search_, run_, submit_) and descriptions present. However, critical gaps exist: (1) Parameter descriptions are minimal or absent for key inputs like 'dir' in run_grep/run_ripgrep, LLMs cannot infer what 'dir' means without explicit guidance. (2) Output schemas are completely undocumented, no tool declares what fields it returns, forcing LLMs to guess structure. (3) Error handling is absent, no guidance on retryability, recovery steps, or what to do when grep/ripgrep fail. (4) The submit_patch tool accepts a raw 'diff' string with no validation hints, format constraints, or examples. (5) No tool annotations (readOnlyHint, destructiveHint) despite clear risk profiles (search tools are read-only; submit_patch is destructive). Baseline: 194 chars avg description, 72 chars avg param annotation, ASTRA descriptions are 50-100 chars, param annotations often missing entirely.
Run grep to search for a pattern in the repository. Returns matching lines.
Run ripgrep (rg) to search for a pattern in the repository. Faster and more expressive than grep.
Search the codebase using ASTRA semantic search. Returns the most relevant code symbols and snippets.
Submit the final unified diff patch to fix the issue. This ends the session. Call this only when you are ready to submit your fix.
Output schemas completely undocumented. No tool declares what fields it returns (e.g., does search_astra return {results: [{file, line, symbol}]} or {matches: [...]}?). LLMs cannot plan downstream calls or extract data without knowing response structure.
Parameter 'dir' in run_grep and run_ripgrep lacks description. LLMs cannot infer whether 'dir' is a relative path, absolute path, or glob pattern. Description should state: 'Subdirectory within the repository to search in (default: . = root). Use relative paths like src/main or . for root.'
No error handling guidance. Tools do not document what happens on failure (pattern not found, permission denied, timeout). Error responses should guide recovery: 'Pattern not found. Try a simpler regex or check the directory path.'
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 49 | <=2025-11-25 | v2 |
submit_patch tool accepts raw 'diff' string with no format validation or examples. Description should specify: 'Unified diff format with --- and +++ prefixes, starting with file headers. Example: --- a/file.rs\n+++ b/file.rs\n@@ -10,5 +10,6 @@'
No tool annotations despite clear risk profiles. search_astra, run_grep, run_ripgrep should declare readOnlyHint=true. submit_patch should declare destructiveHint=true and idempotentHint=false. Annotations help agents reason about side effects and retry safety.