Loading ToolBench
Server provides 7 tools with reasonable naming and descriptions. All tools have explicit schemas with typed parameters and descriptions. However, multiple quality gaps prevent a higher score: (1) Output schemas are not formally documented, responses are described narratively but lack JSON Schema definitions. (2) Error handling is minimal, tools return empty dicts or generic success/failure without actionable recovery guidance. (3) Some parameter descriptions lack format constraints or validation guidance. (4) Tool descriptions are functional but not optimized for LLM selection, they lack WHEN-to-call context. Tool names follow verb_noun convention (list_*, get_*, search_*, sync_*) which is solid.
Return the top-level ro-crate-metadata.json content for the given crate_id. This is a convenience endpoint to retrieve the main metadata without downloading and extracting the entire crate.
Perform a keyword search over indexed crates using a simplyfied lucence search syntax.
List indexed RO-crates with pagination support
Return searchable fields from the configured store (entry columns and materialized entity properties).
Perform a semantic search over indexed crates using the given query string.
List raw storage resources from the configured backend (read-only).
Output schemas not formally documented. Tools return dict[str, Any] with no JSON Schema definition of response structure. Response fields (count, meta, success, status, message) are inferred from implementation but not declared for LLM planning.
Error handling lacks actionable recovery guidance. sync_index_with_storage returns {"status": "error", "message": "Indexer not configured"} with no suggestion for what to do next. get_crate_metadata returns empty dict {} on failure, giving the LLM no signal that an error occurred.
Description lacks WHEN-to-call context. semantic_search and keyword_search have very similar descriptions ('Perform a [semantic|keyword] search...') but no guidance on which to use and when. LLMs will struggle to disambiguate.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 59 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 0 | - | v1 |
Trigger a synchronization of the index with the storage backend. This will scan the storage backend for new or removed crates and update the index accordingly. This operation may take some time depending on the number of resources in storage.
Parameter constraints not explicitly stated in descriptions. storage_list_resources accepts suffixes as array with default [".zip"] but description does not explain valid file extensions, expected format, or what happens if an invalid suffix is provided.
Tool description for sync_index_with_storage mentions "may take some time" but does not specify typical duration, timeout behavior, or if this is a long-running operation that should be treated differently by the agent.