Experiment queue scheduler and figure specification renderer for academic paper automation
Critical gaps across all dimensions. Tool descriptions range from adequate to vague; most lack actionable context for LLM selection. Input schemas present but sparse, many parameters lack type constraints and descriptions. No output schemas documented. Error handling absent. Tool names lack clear action verbs (e.g., 'queue_manager' is a noun, not verb_noun). No parameter validation guidance. Security concerns: queue_manager and paperreview tools accept file paths without sanitization hints. Composition issues: tools operate on disparate domains (experiment scheduling, figure rendering, paper review, proteomics) with no clear chaining. Per-tool scores: build_manifest=38, queue_manager=28, figure_renderer=35, validate_spec=32, paperreview_submit_and_wait=35, paperreview_render=30, analyze_region_proteome=18.
Compute reproducible summaries for the Codex zebrafish brain paper. Analyzes proteome data, computes statistical metrics, and generates CSV/JSON outputs.
Convert grid specs into queue_manager manifest.json. Reads a YAML or JSON grid specification file and expands it into a manifest with phases and jobs using Cartesian product over grid parameters.
ARIS FigureSpec to SVG Renderer. Converts a FigureSpec JSON into publication-quality SVG for academic papers. Deterministic: same spec produces same SVG every time.
Render markdown and scorecard from an existing review JSON without resubmitting to paperreview.ai.
Submit a PDF to paperreview.ai for review and wait for the review to complete. Handles S3 upload, confirmation, and polling for review status.
ARIS experiment-queue scheduler. Runs on SSH remote host or locally. Reads a manifest, launches jobs across free GPUs via screen, retries on OOM, cleans stale screens, and writes state continuously to disk.
analyze_region_proteome has empty input schema (no parameters documented). LLM cannot determine what inputs are required or valid.
Tool names lack action verbs. 'queue_manager', 'figure_renderer', 'validate_spec' are nouns or gerunds, not verb_noun pattern (e.g., 'manage_queue', 'render_figure', 'validate_figure_spec'). LLMs infer intent from names, these are ambiguous.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields. E.g., build_manifest output structure, queue_manager state format, figure_renderer SVG metadata, all undocumented.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 38 | 2026-07-28+ | v2 |
Validate FigureSpec JSON structure and content. Returns list of validation issues including critical errors and warnings.
File path parameters (config, output, spec_file, pdf, review_json) lack sanitization guidance. No mention of path traversal validation, symlink handling, or permission checks. Tools accept arbitrary paths without documented constraints.
Error handling absent. No guidance on retryability, user-fixable vs fatal errors, or recovery steps. E.g., queue_manager OOM retry logic exists in code but not documented in tool description. LLM cannot reason about failure modes.