Loading ToolBench
Four tools with complete schemas and descriptions. Naming follows verb_noun pattern (list_, describe_, get, count). Descriptions are clear and domain-specific (10 - 200 chars, within baseline). All parameters have types and descriptions. However, parameter descriptions lack actionable constraints (e.g., no enum for entity names, no format hints for where clauses). Output schemas are not documented, LLMs cannot predict response structure. Error handling is present but generic. No tool annotations (readOnlyHint, idempotentHint). Missing guidance on when to call discovery tools vs. direct queries.
Count CiviCRM records matching a where clause. Cheaper than civicrm_get for 'how many' questions.
Return field metadata, available actions and query hints for a single CiviCRM entity. Use the pseudoconstant fields to query by human-readable name (e.g. ['status_id:name','=','Current']).
Query CiviCRM records via APIv4 get. Returns {count, values}. Use pseudo-constant suffix names in where clauses (e.g. 'status_id:name') for human-readable queries. Use dot-notation in select for joins (e.g. 'contact_id.display_name').
List CiviCRM entities available on this site. Call once per session before guessing entity names.
Output schemas not documented. LLMs cannot predict response structure (count, values, field types). Forces trial-and-error parsing.
Parameter descriptions lack actionable constraints. 'entity is required' does not tell LLM which entities are valid. 'where clauses' lacks format/syntax guidance. No enums, patterns, or examples.
No tool annotations (readOnlyHint, idempotentHint). All four tools are read-only but this is not machine-readable. Agents cannot distinguish safe tools from destructive ones.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 68 | <=2025-11-25 | v2 |
Discovery tool guidance missing. Description of civicrm_list_entities says 'Call once per session' but does not explain when to call civicrm_describe_entity vs. direct civicrm_get. Agents may skip discovery.
Error handling is generic. No recovery guidance. If civicrm_get fails with 'invalid entity', LLM does not know to call civicrm_list_entities. No categorization of retryable vs. user-fixable errors.