Loading ToolBench
A collection of scaffolding and utility scripts for trapstreet task and solution development. Provides tools for validating task provenance, scaffolding solution directories, and scaffolding task directories with proper structure and validation.
Four tools with complete input schemas and descriptions, but quality varies significantly. check_provenance and validate_task have clear, actionable descriptions (120-180 chars). scaffold_solution and scaffold_task descriptions are longer (200+ chars) but lack specificity about outputs and error cases. All tools have typed parameters with descriptions, meeting baseline requirements. However, no output schemas are documented, and error handling guidance is minimal. No tool annotations (readOnlyHint/destructiveHint) despite clear risk profiles. Parameter descriptions are adequate but lack format constraints and validation rules.
Verify a task's local git HEAD matches what's actually published on trapstreet.run before running/submitting a solution against it. Prevents submission failures due to provenance mismatch.
Scaffold a trap-cli solution directory with shared solution.py and trap.yaml files for one or more model/provider variants. Generates directory structure, environment files, and template code.
Scaffold a new trapstreet task directory with gold.cases.json, build_cases.py, judge.py, grader.py, traptask.yaml, tests/, and README.md. Provides mechanical parts every task shares with customization points for task-specific logic.
Self-consistency checks for a task directory. Validates that build_cases.py runs clean, traptask.yaml matches gold.cases.json, case ids are opaque, no answer leaks into inputs/, and judge.py survives malformed input.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract structured data without knowing return types.
Missing tool annotations despite clear risk profiles. scaffold_solution and scaffold_task are WRITE operations; check_provenance and validate_task are READ_ONLY. Annotations enable agents to reason about side effects.
Parameter descriptions lack format constraints and validation rules. E.g., 'task_slug' and 'repo_path' have no guidance on valid formats, length limits, or character restrictions.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 51 | 2026-07-28+ | v2 |
Error handling descriptions are absent. Tools do not explain what errors can occur, whether they are retryable, or what the LLM should do next (e.g., 'If task not found, try searching available tasks').
scaffold_solution and scaffold_task descriptions are verbose (200+ chars) and lack clarity on what is returned or how to use outputs. Descriptions should state WHAT, WHEN, and WHAT IS RETURNED.