Loading ToolBench
Reference implementation MCP server for the DERIVA platform
Server has 25 tools with consistent naming (verb_noun pattern) and basic descriptions. All tools have input schemas with type definitions and parameter descriptions. However, descriptions are often generic (10-50 chars) and lack actionable context for LLM selection. Output schemas are not documented. Error handling guidance is absent. Security considerations (permission gates, audit trails) are not visible in tool definitions. RAG tools lack clarity on when to use each variant (rag_ingest_docs vs rag_ingest_datasets). Prompts are registered but tool annotations (readOnlyHint, destructiveHint) are missing despite clear risk levels.
Retrieve annotations for a catalog entity
Guide prompt for using annotation tools
Set or update annotations for a catalog entity
Guide prompt for using catalog introspection tools
Introspect a DERIVA catalog schema
Create new entity rows in a DERIVA catalog
Delete entity rows from a DERIVA catalog
Fetch entity rows from a DERIVA catalog
Tool descriptions are generic and lack actionable context. Most descriptions (10-50 chars) do not explain WHEN to use the tool vs similar alternatives, what prerequisites exist, or what the LLM should expect in the response. Example: 'Fetch entity rows from a DERIVA catalog' does not explain the ERMrest path format, pagination behavior, or when to call entity_get vs catalog_introspect.
Output schemas are not documented. Tools return data but the response structure is not visible in the tool definition. LLMs cannot plan downstream calls or extract required fields (e.g., does entity_get return entity_id for use in entity_update?). This forces agents to guess or make discovery calls.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 55 | <=2025-11-25 | v2 |
Guide prompt for using entity CRUD tools
Update existing entity rows in a DERIVA catalog
Download a file from DERIVA Hatrac object store
Upload a file to DERIVA Hatrac object store
Guide prompt for using query tools
Import pre-computed chunks into the RAG vector store
Index a catalog schema into the RAG vector store
Enrich and index dataset rows using a registered dataset enricher
Ingest documentation sources into the RAG vector store
List all registered RAG documentation sources
Search the RAG vector store for documentation and schema information
Get RAG subsystem status and configuration
List available resources exposed by the server
Retrieve schema definition from a DERIVA catalog
Get the status of a long-running task
List all tasks for the current user
Retrieve vocabulary/controlled vocabulary from a DERIVA catalog
Tool annotations (readOnlyHint, destructiveHint, idempotentHint) are missing despite clear risk levels declared in the tool list. entity_delete is marked DESTRUCTIVE but has no annotation. entity_create/update are WRITE but lack idempotentHint guidance. This prevents clients from applying appropriate safeguards (confirmation dialogs, dry-run modes).
RAG ingest tools lack clarity on mutual exclusivity and use cases. rag_ingest_docs, rag_ingest_datasets, and rag_import_chunks all populate the vector store but their distinctions are unclear. Descriptions do not explain when to use each (e.g., rag_ingest_datasets requires an 'indexer_name' but no guidance on how to discover valid indexer names).
Error handling and recovery guidance are absent from tool definitions. No tool description explains what errors are retryable, what user-fixable errors look like, or what the LLM should do if a call fails. Example: entity_delete has no guidance on cascade behavior, permission errors, or referential integrity violations.