Loading ToolBench
AI Parliament-driven systematic literature review automation. Provides end-to-end research pipeline tools for paper discovery, screening, downloading, AI-powered review, and report generation.
FWMA has 14 tools with basic descriptions and parameter schemas, but significant gaps prevent higher scores. Descriptions are present but generic, many under 100 chars and lacking WHEN/WHY context. No output schemas documented. Error handling is absent, tools return dicts with no guidance on failure modes or recovery. Tool names are reasonable (verb_noun style) but some are vague ('run_create' should be 'create_run'). Async job handling (download, review, report, writing_review) returns job_id but no guidance on polling or timeout behavior. No input validation hints, no enum constraints for threshold/format params. Security: no mention of API key injection, rate limiting, or permission gates.
Read a text artifact (JSON, Markdown, log) from a research run.
Check citation reasonability in a manuscript against cited papers.
Crawl papers from configured sources (OpenAlex, arXiv, OpenReview). Returns crawl statistics and paper count.
Download PDFs for screened papers. Long-running operation with progress reporting. Uses multi-strategy fallback: direct URL → Unpaywall → DOI redirect → browser.
Get status for an asynchronous job by job_id.
Run a standalone AI Parliament debate on any topic. Useful for general-purpose multi-agent evaluation and discussion.
Parameter descriptions missing or minimal. 'sources' in run_create, 'bib_index' in citation_check, 'ctx' in download/review lack detail on structure, format, or constraints. Rubric: 100% of A+ tools have param descriptions.
No output schemas documented. Tools return dicts but LLMs cannot plan downstream calls without knowing field names, types, and structure. Rubric: 100% of A+ tools have documented return types.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 54 | 2026-07-28+ | v2 |
Extract tables, figures, and formulas from PDF using vision model.
Generate research summary report from all reviews. Format options: markdown, json, both.
Review papers with AI Parliament debate. Long-running operation. Each paper goes through structured multi-agent debate: Chair opens → Members argue → Vote check → Final verdict with score.
Create a new research run with requirement and source configuration. Returns run_id and initial status.
Get status and available artifacts for a research run.
Screen papers for relevance using AI. Threshold options: high_only, high_medium, all_selected. Returns screening statistics.
Generate AI-powered search strategy from a research requirement. Returns suggested sources configuration for OpenAlex, arXiv, and OpenReview.
Review manuscript writing quality using AI Parliament debate. Returns verdict, debate log, and detailed improvement suggestions.
Enum constraints missing for string parameters. 'threshold' in screen accepts 'high_only|high_medium|all_selected' but no enum defined. 'format' in report accepts 'markdown|json|both' but no enum. Free-form strings invite hallucinated values.
Async operations (download, review, report, writing_review) return job_id but no polling guidance, timeout behavior, or error recovery hints. LLMs cannot determine when to check status or how to handle hung jobs.
Tool naming inconsistency: 'run_create' should follow verb_noun convention as 'create_run'. 'pdf_vision' is vague, 'extract_pdf_content' or 'analyze_pdf_vision' is clearer. Inconsistent naming increases LLM confusion.