Loading ToolBench
TypeScript SDK for typeship. Resolve an OpenAPI or GraphQL Definition, diagnose it, and keep every selected CLI, MCP, and SDK Target current.
This is a TypeScript SDK wrapper, not an MCP server. The evaluation treats it as if it were exposed as an MCP tool set. Tool naming follows verb_noun convention (retrieve, list, update, run) and is generally clear. Descriptions are present and substantive (avg ~150 chars), exceeding the 10-char minimum. However, input schemas are minimal, most tools accept only 1-2 parameters with basic type info (string, number). Output schemas are not documented in the source. Error handling is absent from tool definitions. The SDK itself has good HTTP client infrastructure (retries, timeouts, validation), but this is not reflected in the tool interface. Pagination is mentioned in descriptions but not formalized in schema. No tool annotations (readOnlyHint, destructiveHint) are visible.
The account behind the presented credentials. Returns the account associated with your credential. The Typeship CLI uses this endpoint for `whoami`.
List API keys. Lists key metadata and the last four characters of each key. Full keys are not returned. Create keys in the Console. Auto-paginates: `for await (const item of …)` walks every page.
Revoke an API key. Revokes a key. Repeating the request returns the same result. With OAuth, members can revoke their own keys; organization admins can revoke any key. Organization API keys can revoke any key in their account.
List Definition Revisions. Lists the Definition's revisions, newest first. Source content is not included; retrieve the revision content or individual documents separately. Auto-paginates: `for await (const item of …)` walks every page.
Retrieve a Definition Revision. Returns metadata for a saved Definition Revision. Retrieve its resolved content or individual source documents separately.
Output schemas not documented. Tool descriptions mention what is returned (e.g., 'Returns the account associated with your credential') but do not specify the JSON structure, field names, or types. LLMs cannot plan downstream tool calls or extract specific fields without knowing the response schema.
Minimal input schemas. Most tools accept only 1-2 parameters with type declarations (string, number) but no constraints, enums, or ranges. E.g., 'limit' is typed as number but lacks min/max bounds; 'cursor' is a string with no format guidance. Unbounded parameters invite invalid LLM inputs.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). The Risk field in the metadata (READ_ONLY, WRITE, DESTRUCTIVE) is not exposed in the tool schema. LLMs cannot determine which tools are safe to retry or which modify state without explicit annotations.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 52 | 2026-07-28+ | v2 |
Retrieve a Definition Revision's canonical content. Returns the saved, resolved content for this revision. Save it locally or compare it with another revision.
Retrieve one source document from a Definition Revision
Retrieve a Definition
Update and resolve a Definition. Resolves the source documents and records a new Definition Revision before saving the update. A `Idempotency-Key` UUID is generated per call (stable across retries) unless you pass one.
Generate one package from a Definition. Returns one generated package without creating a Project. Supports idempotent retries; keyed responses include generated files in the replay cache. Anonymous and Free requests include the first 25 operations. Paid plans include all operations. Anonymous requests are rate limited by IP address. Check `limits` for omitted operations; an invalid API key returns `401`. An anonymous URL request without source headers may return `claim.url`. Sign in through that link within seven days to save the recipe as a Project. A `Idempotency-Key` UUID is generated per call (stable across retries) unless you pass one.
Retrieve a generation. Returns the Generation result. Successful results include files, or a file index when the package is too large to inline.
Fetch one file from a generation. Returns one file's raw content. Use a path from `files_index` when the Generation reports `files_omitted: true`.
Error handling not documented. Tool descriptions do not explain what errors can occur, how to recover, or what the LLM should do next. E.g., 'apiKeys.revoke' says 'Repeating the request returns the same result' but does not document failure modes (invalid key ID, permission denied, etc.).
Pagination formalized in descriptions but not in schema. 'apiKeys.list' and 'definitionRevisions.list' mention 'Auto-paginates' and describe cursor/limit behavior in prose, but the schema does not declare a return type with next_cursor or total_count fields. LLMs cannot reliably iterate without explicit pagination schema.