Loading ToolBench
Provider-neutral OpenDeepThink CLI and MCP server. Population-based reasoning via pairwise judging and Bradley-Terry ranking.
DeepThonk provides 17 tools with complete JSON schemas and descriptions for all parameters. Naming follows verb_noun convention (plan, start, status, result, cancel, rank, mutate, resume, export, profile_*). However, descriptions are often terse (10-50 chars) and lack actionable context for LLM selection. Many tools expose complex nested objects (model_output_tokens, critique_limits, rank, concurrency, prompts) with minimal guidance on their structure or when to use them. Error handling and recovery guidance are absent. Tool composition is reasonable but some tools (deepthonk.start vs deepthonk.run) have overlapping responsibilities without clear differentiation.
Request cancellation for a running job by writing cancel.json in the run directory.
Export a run summary or trace.
Inspect run.lock and return its exact fingerprint without mutating it.
Reclaim run.lock only when its current bytes match the inspected fingerprint exactly.
Mutate one supplied candidate with critique.
Estimate model calls for a DeepThonk profile.
Delete a saved profile.
Complex nested object parameters (model_output_tokens, critique_limits, rank, concurrency, prompts) lack internal schema documentation. LLMs cannot infer the structure of these objects, forcing trial-and-error or external documentation lookup.
Descriptions for deepthonk.start and deepthonk.run are identical and generic ('Start DeepThonk...' vs 'Run DeepThonk...'). No guidance on when to use start (background) vs run (blocking), or how they differ in behavior and output.
No error handling or recovery guidance in any tool description. Tools like deepthonk.start and deepthonk.run can fail (invalid config, API errors, budget exceeded) but descriptions do not indicate what to do on failure or which errors are retryable.
Inferred effective spec: 2025-06-18+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 55 | 2025-06-18+ | v2 |
List saved profile names.
Save a profile bundle.
Show details of a saved profile.
Rank supplied candidates with pairwise judging and Bradley-Terry aggregation.
Replace only legacy [redacted] numeric budget fields with explicit original values.
Return final summary/resource URIs when a background job is complete.
Detect resume state (default) or replay an interrupted run with continue: true.
Run DeepThonk through the shared core engine.
Start DeepThonk in the background and persist status in the run directory.
Read persisted job/run status from a run directory.
Sampling parameters (sampling_model_hints, sampling_cost_priority, sampling_speed_priority, sampling_intelligence_priority) are present but Sampling is a deprecated MCP feature (removed ~2027-07-28). These should be migrated to direct LLM provider integration or removed.
Lock management tools (deepthonk.lock_inspect, deepthonk.lock_reclaim) expose low-level file-system semantics (SHA256 fingerprints, run.lock files) that are implementation details. LLMs cannot reason about lock ownership or fingerprint matching without domain expertise.