Loading ToolBench
An autoresearch-style AI operating system with experiment tracking, open-loops management, wiki linting, and brand-aware HTML rendering tools. Designed for AI-driven research workflows with deterministic evaluation harnesses.
gAIOS provides 12 tools with explicit schemas and descriptions, but quality is inconsistent. Naming follows verb_noun convention (experiment_log, graphify_setup, open_loops, render_brand_html, wiki_lint), which is good. However, descriptions vary widely in clarity and actionability. Most parameter descriptions are present but generic. No output schemas documented. Error handling is minimal, tools return exit codes without recovery guidance. The experiment_log and open_loops tools are well-structured with clear TSV/file semantics, but graphify_setup and render_brand_html lack depth. No tool annotations (readOnlyHint, destructiveHint, idempotentHint) present. Overall, this is a functional but mediocre toolkit, typical of community servers.
Log a single experiment result to the TSV file with id, metric value, status (keep|discard|crash), and optional note
Find and display the best experiment (by metric) from the kept experiments. Returns the winner with lowest (or highest if --maximize) metric value.
Initialize a new experiment results TSV file with header row (id, metric, status, note)
Display all logged experiments from the TSV file in a formatted table
Check whether the graphify CLI is available on PATH and the graphify Python module is importable. Exit 0 if present, 1 if not.
Install graphify CLI (if missing) via uv, pipx, or pip, then wire the /graphify skill into each requested platform (claude, codex). Idempotent.
Add a new open loop (commitment) to the tracking file with optional owner and due date
No output schemas documented. Tools return results but LLMs cannot predict structure (e.g., what fields does 'show' return? Is it a table, JSON, or text?). This forces agents to guess and wastes tokens parsing unstructured output.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). LLMs cannot distinguish safe reads from destructive writes without explicit hints. experiment_log add and open_loops done are destructive but unmarked.
Error handling is minimal. Tools return exit codes (0/1) with no recovery guidance. 'graphify_setup check' exits 1 if missing, but does not tell the LLM to call 'graphify_setup install'. Errors should guide next steps.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 51 | 2026-07-28+ | v2 |
Mark an open loop as closed by its numeric id
List all open loops (or all including closed if --all flag is set)
List all open loops that are overdue as of today (or a specified date). Exit code 1 if any overdue loops exist.
Render an on-brand HTML document from a content JSON, styled deterministically from brand-tokens.json. Supports headings, paragraphs, KPI cards, tables, and callouts.
Lint and leakage-scan the committed wiki/ directory before commit. Detects broken relative links, IBANs, secrets/API keys (errors), and currency amounts, orphan pages, PHI-review markers (warnings).
graphify_setup install description is vague ('Install graphify CLI (if missing) via uv, pipx, or pip, then wire the /graphify skill...'). Does not explain what 'wiring' means, what platforms are available, or what happens if already installed. Lacks actionable detail.
render_brand_html and wiki_lint descriptions are generic and lack context. render_brand_html does not explain what 'on-brand' means or what brand-tokens.json contains. wiki_lint does not clarify what 'leakage-scan' detects or why it matters.