Loading ToolBench
LLM-powered logical reasoning with Prolog - Calculator for logic
The server defines 5 tools with reasonable schemas and descriptions. Tool naming follows verb_noun patterns (execute_prolog, list_rule_bases, get_rule_base, save_rule_base, delete_rule_base). Descriptions are present and moderately detailed (90-250 chars), exceeding the 20-char floor. Input schemas are visible with type declarations and parameter descriptions. However, several gaps reduce overall quality: (1) Output schemas are not explicitly documented in the code, responses are dict objects without formal return type annotations visible in the definitions; (2) No error handling guidance in descriptions, tools return dict with success/error fields but descriptions don't explain error cases or recovery paths; (3) Parameter descriptions lack constraint details (e.g., max_results defaults to 100 but no min bound stated); (4) Destructive operations (delete_rule_base) lack confirmation/dry-run patterns; (5) Response field naming is inconsistent (some return 'success', others may return different structures). These gaps align with a 'Fair' (C/low-60s) assessment.
Delete a saved rule base by name.
Execute Prolog code and return reasoning results. Write Prolog facts and rules, then run a query against them. Supports CLP(FD) constraints, negation-as-failure, and all standard SWI-Prolog features.
Retrieve the Prolog source of a saved rule base.
List all saved rule bases with description and tags. Returns ``{"rule_bases": [{"name": str, "description": str, "tags": list[str]}, ...]}`` sorted by name. Metadata is extracted from the leading ``% description:`` / ``% tags:`` comments of each rule base file (see §4.10).
Save a named rule base containing Prolog rules that can be reused across ``execute_prolog`` calls. Use this for stable, reusable knowledge (e.g. ``piece_moves`` for chess piece movement rules). For one-time facts, include them directly in ``prolog_code`` instead.
Output schemas not documented in tool descriptions. Responses are returned as dicts (e.g., from execute_prolog.model_dump()), but the structure and field types are not formally declared in the tool definition. LLMs cannot reliably plan downstream operations without knowing what fields to expect.
Error handling lacks recovery guidance. Tool descriptions mention success/error dict fields but do not explain: (a) when errors occur, (b) what values are retryable vs. user-fixable, (c) what the agent should do next. E.g., execute_prolog can fail on rule base load (RULEBASE_001/002) but description doesn't guide recovery.
Destructive operation (delete_rule_base) lacks confirmation/dry-run pattern. The tool directly deletes a rule base with no warning, undo capability, or confirmation step. Agents can accidentally destroy data without a safety mechanism.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 57 | <=2025-11-25 | v2 |
Parameter constraints not fully specified. execute_prolog's max_results defaults to 100 but description does not state minimum (presumably 1) or maximum bounds. Unbounded integer parameters risk LLMs passing absurd values. No constraints on prolog_code or query length.
Response field naming inconsistency. save_rule_base and delete_rule_base descriptions do not document their return structures, but code hints at {'success': bool, 'error': str, ...}. Without formal schema declaration, LLMs cannot parse responses reliably or chain tools.
get_rule_base description is vague ('Retrieve the Prolog source of a saved rule base'). Does it return the full source? In what format? As a string field? As structured data? Description should explicitly state the response structure and format.