Loading ToolBench
Astra demonstrates strong naming conventions (verb_noun pattern: astra_semantic_search, astra_get_callers) and comprehensive descriptions (avg ~150 chars, well above 34-char minimum). All 21 tools have clear, actionable descriptions explaining WHAT, WHEN, and WHY. Input schemas are present and typed for all tools. However, output schemas are not documented in the visible code, LLMs cannot predict return structure. Error handling is minimal; no recovery guidance or categorization. Parameters lack some constraints (e.g., limit/threshold ranges not specified). Tool composition is excellent (single responsibility, clear chaining via path/target/query params), but output documentation gaps and missing error patterns prevent A-grade scoring.
Select tests affected by changed paths.
Index if needed, then extract dependency-complete context around a query.
Generate a read-only pytest scaffold for a target.
Find callers from the graph, indexing first when artifacts are missing.
Rank declarations by graph and AST risk.
Return one architecture-readiness decision for CI and pull requests.
Index if needed, then combine semantic matches with structural expansion.
Output schemas not documented. LLMs cannot predict return structure (fields, types, pagination). Breaks downstream tool chaining and forces agents to guess field names.
No error handling or recovery guidance. Tools return failures without actionable next steps (e.g., 'index missing' should suggest astra_index_repo). Agents cannot self-correct.
Parameter constraints missing. 'limit' and 'threshold' parameters lack min/max bounds. LLMs may pass absurd values (limit=999999, threshold=-100) causing timeouts or logic errors.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 62 | 2026-07-28+ | v2 |
Calculate the upstream blast radius of a declaration.
Index or refresh a local repository before using other Astra tools.
Check continuous indexing status.
Find a graph path, indexing first when this workspace has no graph artifact.
Preview a graph-ordered structural rename.
Run the graph-selected impacted tests.
Search indexed chunks; automatically index first when artifacts are missing.
Rank highly depended-on declarations by incoming links and PageRank.
Start continuous background indexing for a workspace.
Stop continuous background indexing.
Map source declarations to tests that reach them.
Check cycles, orphan declarations, and high fan-out.
Orchestrate impact, risk, test selection, scaffolding, and validation.
Render the current Astra graph and vector artifacts.
Destructive tools (astra_run_impacted, astra_validate_change, astra_start_watch, astra_stop_watch) lack confirmation or dry-run support. No idempotent hints or destructiveHint annotations.
Tool annotations missing. No readOnlyHint, destructiveHint, or idempotentHint metadata. Agents cannot distinguish safe read-only tools from risky write operations without parsing descriptions.