Loading ToolBench
Deterministic data layer for agents — a hand-rolled stdio JSON-RPC 2.0 MCP server with zero MCP-SDK dependency, exposing eight verbs (catalog, search, get, resolve, estimate, export, jobs, verify) as typed MCP tools
Pinakes exposes 8 tools with explicit schemas and descriptions visible in handler.go and gen_tools.json. All tools have non-empty descriptions (range 80 - 180 chars, within baseline 34 - 392). Input schemas are present with typed parameters and descriptions. However, critical gaps reduce quality: (1) Tool names lack action verbs, 'pinakes_catalog', 'pinakes_search', 'pinakes_get' are prefixed with domain name rather than verb-first (list_sources, search_records, get_records). (2) Descriptions are functional but generic; they explain the internal five-step pipeline rather than user intent. E.g., 'pinakes_search' says 'Runs the full five steps with Operation=search: validate, coerce, Resolve, translate', an LLM cannot infer when to call this vs pinakes_get without domain knowledge. (3) Output schemas are not documented in the visible code; only input schemas are shown. (4) Error handling is minimal, no recovery guidance, no categorization of retryable vs fatal errors. (5) No tool annotations (readOnlyHint, destructiveHint) despite clear risk levels (7 READ_ONLY, 1 WRITE). Average per-tool score: 52.
Lists every source's metadata, or — when a SourceID is given — that one source's filter schema
Returns the connector's dry-run Plan estimates (Operation='estimate') WITHOUT retrieving. It is the documented exception to step 4: instead of Resolve it calls the additive resolver.Estimate
Materializes a query result (Operation='export'). At Foundation time the export is SYNCHRONOUS: it Resolves the query and returns the manifest with an inline local Location and an empty JobID
Fetches explicit records by identifier (Operation='get'). It is id-driven so there is no validate-vs-catalog step; it coerces and Resolves
Queries the status of async export jobs or cancels a job in progress
Maps identifiers into a target namespace (Operation='resolve'). Like Get it is id-driven (no filter validation); it coerces and Resolves
Runs the full five steps with Operation='search': validate the filters against the source catalog, coerce to a NormalizedQuery, Resolve, translate
Tool names lack action verbs. All 8 tools are prefixed 'pinakes_' (domain name) rather than starting with verb (list_, search_, get_, create_, etc.). LLMs infer intent from the first word, 'pinakes_catalog' does not signal 'list' or 'describe'. Rename to list_sources, search_records, get_records, resolve_identifiers, estimate_query, export_results, query_jobs, verify_manifest.
Descriptions explain internal pipeline (five-step Operation, coerce, Resolve, translate) rather than user intent. E.g., pinakes_search: 'Runs the full five steps with Operation=search: validate the filters against the source catalog, coerce to a NormalizedQuery, Resolve, translate'. An LLM cannot determine when to call this vs pinakes_get without domain expertise. Rewrite to: 'Search a data source using filters. Returns matching records. Use this to find records by field values; use get_records to fetch by ID.'
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 46 | 2026-07-28+ | v2 |
Verifies a manifest against the engine's contract: decodes the base64-encoded manifest JSON, unmarshals it to a *contract.Manifest, and calls resolver.Verify
Output schemas not documented. Only input schemas are visible in the code. LLMs need to know what fields to expect in responses so they can plan downstream calls and extract data. Add output schema documentation for all 8 tools, e.g., 'Returns: {records: [{id, fields}], total_count, next_cursor}'.
No tool annotations despite clear risk levels. 7 tools are READ_ONLY, 1 is WRITE (pinakes_jobs with cancel flag). MCP spec supports readOnlyHint and destructiveHint annotations. Add these to help agents reason about side effects and retry safety. pinakes_jobs should be marked destructiveHint=true when cancel=true.
Error handling lacks recovery guidance. No visible error categorization (retryable vs user-fixable vs fatal) or actionable messages. E.g., if a source_id is invalid, the error should suggest 'Call list_sources to see available sources' rather than a bare 404. Implement pattern:recovery-guide for all tools.