Loading ToolBench
MCP server that lets coding agents generate datasets, fine-tune, and evaluate LLMs.
TuneForge has well-structured tool schemas with proper JSON Schema definitions and clear naming conventions. All 12 tools start with action verbs (generate, train, merge, evaluate, list, get, cancel, wait, estimate, health). Descriptions are present and substantive (avg ~150 chars), explaining WHAT each tool does and WHEN to use it. However, parameter descriptions are sparse or missing entirely, most params lack explanatory text beyond the name. Output schemas are not documented, forcing LLMs to infer result structure. Error handling is minimal; no recovery guidance or actionable error messages visible. Security considerations (secret injection, permission gates) are absent from the tool definitions.
Request cooperative cancellation of a running job. The worker checks the cancellation flag at every progress tick and exits cleanly. Returns the job state after the request.
Estimate VRAM needed for an SFT run on a given base model BEFORE starting training. Returns a diagnostic with estimated need vs available VRAM.
Evaluate a base model (optionally with an adapter) on a JSON dataset using an LLM-judge. Returns average score and per-example details.
Generate a supervised fine-tuning dataset from a product description and optional source text (e.g. PDF extract). Uses the teacher model for synthesis and an LLM-judge for quality filtering. Runs as a background job.
Fetch the current status, progress, and (if finished) result or error for a job.
Report server health: workspace location, Ollama reachability, model count.
Parameter descriptions are missing or minimal. Most params (e.g., 'base_model', 'dataset_path', 'epochs', 'learning_rate') have no description text, only type and default. LLMs cannot infer what 'base_model' means (HuggingFace ID? local path? model name?) without explicit guidance.
Output schemas are not documented. Tools return job_id, status, results, or diagnostics, but the response structure is not specified. LLMs must guess what fields to expect, risking failed downstream calls.
Error handling is not visible in tool definitions. No guidance on retryable vs fatal errors, no actionable error messages, no recovery suggestions. If a job fails or VRAM estimation fails, the LLM has no recovery path.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 56 | <=2025-11-25 | v2 |
List recent jobs. Optionally filter by kind or status.
List models currently available from the configured Ollama endpoint.
Merge a LoRA adapter into the base model and save a standalone Transformers checkpoint.
Run reward-model-free policy gradient on top of an SFT adapter using an Ollama-hosted teacher as LLM-judge. Returns a job_id.
Start a LoRA SFT run on the provided dataset and base model. Requires the [train] extras installed. Returns a job_id; poll with get_job_status.
Block until a job reaches a terminal state (succeeded/failed/cancelled) or the timeout expires, then return the final job record. Use this when you want to await a result instead of polling.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Tools like 'cancel_job' and 'train_sft' modify state but are not marked as destructive. LLMs cannot distinguish safe reads from risky writes without explicit hints.
Secrets and credentials are not addressed. Tools accept 'teacher_model' and 'base_model' as strings, but no guidance on whether these are HuggingFace tokens, API keys, or local paths. If credentials are embedded in model names, they risk leaking into logs.