Loading ToolBench
MCP server + CLI for UNWATCHED — video URL in, validated runbook out
Server has 5 well-named tools with complete input schemas and descriptions. Tool names follow verb_noun convention (decode_video, get_job, get_runbook, list_library, wait_for_job). All tools have descriptions (avg 120 chars, within 10-1024 range). Input schemas are properly typed with required fields. However, output schemas are NOT documented, LLMs cannot predict response structure. Parameter descriptions are minimal (e.g., 'Video URL (e.g. YouTube)' lacks format/constraint details). Error handling returns text but lacks recovery guidance or categorization. No tool annotations (readOnlyHint/destructiveHint). Missing pagination guidance for list_library despite potentially large result sets.
Start an UNWATCHED decode job for a video URL. Returns the job_id — use wait_for_job to poll and get_runbook to fetch the validated runbook.
Get the status of a decode job plus its artifact list.
Fetch the runbook content for one video of a completed job. Use get_job to discover the video_id.
List previous decodes of your tenant, optionally filtered by a query string.
Poll a decode job until it reaches a terminal status (completed/partial/failed) or the timeout elapses.
Output schemas not documented. LLMs cannot predict response structure (job fields, artifact list format, runbook content type). Forces trial-and-error parsing.
Parameter descriptions lack format/constraint details. 'Video URL (e.g. YouTube)' does not specify accepted protocols, length limits, or error cases. 'profiles' array lacks enum values or examples of valid profile names.
Error handling returns raw error text without recovery guidance. 'Unknown format: md' tells LLM nothing about valid options. Should return 'Invalid format: got "md", must be one of: md, yaml, json'.
list_library lacks pagination parameters (limit, offset, cursor). No guidance on result limits. If library is large, response could exhaust context window.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 58 | 2026-07-28+ | v2 |
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). decode_video is destructive (starts a job); wait_for_job is idempotent. Annotations help LLMs reason about side effects and retry safety.