Loading ToolBench
Purchase requisitions, budgets, and expense management server for maritime vessel fleet operations
The server provides 12 read-only search and export tools for a maritime procurement and budget system. All tools have descriptions (good) and input parameter schemas with types and descriptions (baseline-compliant). However, several critical gaps prevent a higher score: (1) Tool naming mixes patterns inconsistently, some use verb_noun (universal_purchase_search) but others use get_* and descriptive noun patterns (fleet_procurement_and_budget_information) without clear semantic boundaries. (2) Parameter descriptions lack critical details like ranges, allowed values, and validation rules, e.g., question_no parameters enumerate values (11=budget, 121=activities) in descriptions but use magic numbers that require LLM memorization. (3) Output schemas are documented inline (JSON structure in code) but not formally declared in tool definitions; responses are JSON strings, not structured types, forcing LLM parsing. (4) Error handling is minimal, no recovery guidance, no actionable error messages, no distinction between retryable/fatal errors. (5) Tool composition has redundancy: three parallel 'universal_*_search' tools (purchase, budget, expense) plus three 'get_complete_vessel_*_data' tools plus three 'get_complete_fleet_*_data' tools create cognitive load for LLM selection without clear when-to-use guidance. (6) No pagination metadata or result limits documented in descriptions despite tools returning potentially large datasets. Per-tool analysis reveals 6-7 tools at ~65-70, and 5-6 tools at ~55-60 due to schema and description gaps.
Retrieves fleet management information including fleet purchase logs, purchase activities, and budget overviews.
Export ALL budget records for an entire fleet in JSON format. Steps: 1) Gets vessel IMO list from fleet, 2) Exports all budget data via Typesense, 3) Returns JSON.
Export ALL expense records for an entire fleet in JSON format. Steps: 1) Gets vessel IMO list from fleet, 2) Exports all expense data via Typesense, 3) Returns JSON.
Export ALL purchase records for an entire fleet in JSON format. Steps: 1) Gets vessel IMO list from fleet, 2) Exports all purchase data via Typesense, 3) Returns JSON.
Export ALL budget records for a vessel from the budget typesense collection. Retrieves the complete dataset for comprehensive financial planning, budget variance analysis, or bulk data processing.
Magic number parameters (question_no) require LLM memorization of enum values instead of named constants. Values like 11=budget, 121=activities, 257=log table are scattered across three tools without a canonical reference. This violates the constrained-input pattern and forces LLMs to store and recall opaque mappings.
Tool naming inconsistency creates ambiguity. 'universal_purchase_search' vs 'get_complete_vessel_purchase_requisition_data' vs 'get_complete_fleet_purchases' all query purchase data but with different scopes (search vs vessel vs fleet) not signaled in names. No clear when-to-use guidance. An LLM must reason about scope from descriptions rather than infer from names.
Inferred effective spec: 2026-07-28+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 51 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 46 | - | v1 |
Export ALL expense records for a vessel from the expense typesense collection. Retrieves the complete dataset for comprehensive financial analysis, audit reporting, or bulk data processing.
Export ALL purchase requisitions for a vessel from the purchase typesense collection. Retrieves the complete dataset for comprehensive analysis, reporting, or bulk data processing.
Retrieves vessel budget and expense information including budget overviews, variance analysis, expense summary, and detailed breakdowns.
Retrieves vessel purchase order information including POs with forwarders and overdue PO tracking.
Query the budget database using structured parameters to retrieve and analyze vessel budget and expense data from Typesense.
Query the expense database using structured parameters to retrieve and analyze vessel expense and cost tracking data from Typesense.
Query the purchase database using structured parameters to retrieve and analyze purchase requisition and purchase order data from Typesense.
Parameter descriptions lack critical format and constraint details. 'start_date' and 'end_date' across all tools state '(YYYY-MM-DD)' informally; no JSON Schema format declaration (format: date), no range constraints (must be >= today, <= 5 years ago), no validation error examples. Descriptions should say: 'ISO 8601 date (YYYY-MM-DD), must be in range [2020-01-01, today].'
Output schemas not formally declared in tool definitions. Code shows JSON structure is parsed inline (json.dumps with summary + data fields), but tool registration does not expose output schema to the LLM. LLM cannot plan downstream processing without knowing response structure (e.g., that 'data' is an array vs object, what fields each record contains).
Error handling is minimal and non-actionable. Code returns plain strings like 'IMO number is required' or 'Access denied: Vessel IMO X does not belong to your company.' No recovery guidance. No distinction between retryable errors (timeout, rate limit) vs user-fixable errors (invalid IMO) vs fatal errors (database down). LLM cannot self-correct or choose next step.
Tool descriptions lack when-to-use clarity. 'universal_purchase_search' description says 'Query the purchase database' but does not explain: Is this for keyword search (by vendor, vessel name) or filtering by date range? Should I use this before export tools? What fields can I search? Description is 130 chars, baseline is 194; it is too sparse.
Pagination and result limits not documented. Search tools accept 'page' and 'per_page' (default 50, max 250) per code, but tool descriptions do NOT state this. Export tools return ALL records for a vessel/fleet, no cap, no pagination. A query could return thousands of records, exhausting context and token budget. Descriptions should state 'Returns up to 50 results per page' and 'Results may be large; use pagination or filters.'
Parameter 'query_by' and 'filter_by' in search tools reference Typesense syntax (e.g., 'comma-separated field names', 'Typesense filter syntax') without explaining valid field names or filter operators. An LLM must infer or hallucinate valid values. Description should list available searchable fields: 'Searchable fields: vesselName, vendorOrSupplierName, prDescription, orderDate' and provide filter examples: 'Use filter_by="orderDate:[2024-01-01 TO 2024-12-31]" for date ranges.'