Loading ToolBench
23 tools with mixed quality. Strengths: most tools have descriptions (avg ~150 chars) and structured schemas with typed parameters. Critical weaknesses: 6 tools (mloop_train, mloop_promote, mloop_status, mloop_serve, mloop_sample, mloop_trigger) have empty input schemas `{"type":"object","properties":{}}` with no parameters defined, these score 0 on schema. Descriptions are present but often generic (e.g., 'Train a machine learning model' lacks WHEN/WHY context). No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear WRITE vs READ_ONLY risk labels in metadata. Error handling and recovery guidance absent from descriptions. Parameter descriptions are adequate but lack format/range constraints (e.g., 'Path to MLoop project directory' doesn't specify relative vs absolute, or validation rules).
Run a single, read-only exploratory-data-analysis (EDA) aspect on a dataset and return a structured JSON envelope for feature-engineering decisions. Aspects: profile, correlation, importance (needs a label), outliers, distribution. Never mutates the data file or mloop.yaml.
Automated training pipeline: profile data -> train (with auto-time) -> list experiments. Handles the full training workflow in a single call.
Compare metrics across multiple experiments to identify the best performing model.
One-shot time-series anomaly detection over an entire CSV series (SR-CNN, no training, no MLoop project required). Every point returns an anomaly verdict, score, an expectedValue, control limits (controlLower/controlUpper — the chartable band: normal points lie inside it, scaled by the residual dispersion) and the detection margin (marginLower/marginUpper — the gate behind the verdict: isAnomaly implies the value is outside it, but not the converse). Use this instead of train+predict when analyzing a whole series once.
Evaluate a model's performance on test data. Shows detailed metrics.
6 tools have empty input schemas (no properties defined): mloop_train, mloop_promote, mloop_status, mloop_serve, mloop_sample, mloop_trigger. LLMs cannot infer required parameters or constraints.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear WRITE vs READ_ONLY risk labels in metadata. Agents cannot distinguish safe-to-retry operations from destructive ones without explicit hints.
Descriptions for mloop_train, mloop_promote, mloop_status, mloop_serve, mloop_sample, mloop_trigger are under 40 chars and lack WHEN/WHY context. These fail to guide LLM tool selection.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 59 | 2026-07-28+ | v2 |
Declare which columns are features in mloop.yaml (policy only — never changes data). Exclude columns with drop='a,b', keep only certain columns with keep='a,b' (all others excluded, label always kept), or clear all exclusions with reset=true. Returns a JSON envelope listing the excluded (ignored) columns.
Manage feedback for model predictions. Add feedback, list existing feedback, or view feedback metrics.
Analyze and profile a dataset. Shows column types, statistics, missing values, and more. Can read label column from mloop.yaml configuration.
Initialize a new MLoop project. Creates mloop.yaml, datasets/, models/ directories.
List all experiments and their metrics for a model.
View prediction logs for a model. Shows recent predictions with timestamps and results.
Run predictions using a trained model. Uses production model by default.
Run the data preprocessing pipeline. Executes preprocessing scripts defined in the project configuration.
Declare a preprocessing step in mloop.yaml (policy only — never changes data). Add/replace with set='type[:method]' (+ optional columns), remove with remove='type', or omit both to list the current plan. Returns a JSON envelope where each step reports its leakage category (preFeaturizer/csvStage/unsupportedLeakageWarn) and fold-safety. Statistical fit happens fold-internally at train time.
Get a comprehensive overview of the MLoop project: status, all experiments, and production model metrics in a single call.
Promote an experiment to production
Automated training pipeline: profile data -> train (with auto-time) -> list experiments. Handles the full training workflow in a single call.
Sample data from a dataset
Start a model serving endpoint
Get project status
Train a machine learning model
Trigger a model retraining or update
Validate project configuration and data
Parameter descriptions lack format/range constraints. E.g., 'Path to MLoop project directory' doesn't specify relative vs absolute, or validation rules.
No error handling or recovery guidance in tool descriptions. E.g., mloop_predict doesn't explain what happens if model not found or data file missing.