Loading ToolBench
Robot recording integrity auditing with published precision and recall — and an `unassessable` verdict when it cannot tell. MCP server, fleet catalog, training-data gate.
Baglens demonstrates strong definition quality with comprehensive tool coverage (20 tools), well-structured schemas, and detailed descriptions. All tools have input schemas with typed parameters and descriptions. Tool names follow verb_noun conventions (add_source, list_missions, extract_keyframes). Descriptions are substantive (100-400 chars typical), explaining WHAT the tool does and WHEN to use it. However, some parameter descriptions lack specificity about constraints, ranges, and error conditions. Output schemas are documented via Pydantic models but not explicitly exposed in tool definitions. Error handling guidance is minimal, tools do not describe recovery paths or actionable error messages. No tool annotations (readOnlyHint/destructiveHint) despite clear risk classifications (WRITE vs READ_ONLY marked in metadata).
Register a directory of recordings and start indexing it. Indexing extracts metadata, per-topic health, detected events and a small set of numeric signals, so every later catalog.* and compare.* question is answered from the local database instead of by reopening bags. Poll catalog.index_status. Already-indexed missions are skipped unless `force=true`.
Indexing progress, so you can wait intelligently instead of failing. If `running` is true and `progress` is low, ask a different question first and come back — the answers get better as the corpus fills in.
List indexed missions with filters. Paginated. Filter by robot, tag, presence of a topic, duration, health score or verdict. Use `max_health_score` to find the recordings worth investigating first. If a previous call came back truncated, pass its `continuation_token` back here to get the next page; it carries the offset and page size.
Everything the catalog knows about one mission: topics, health, events, signals. Answered from the index — no bag is reopened. Use health.audit_recording when you need the live detail rather than the cached summary.
Split the corpus and compare a metric across the halves. "What changed after v2.4?" `split_by="tag"` with value_a/value_b compares two labelled groups; `split_by="date"` with `split_date` (YYYY-MM-DD) compares before and after a boundary. Metrics: health_score, gap_count, estimated_dropped, max_jitter_cv, event_count, critical_events, duration_s, message_count.
Tool annotations missing: WRITE tools (catalog.add_source, export.trim_bag, export.plot, export.foxglove_layout, export.report) lack destructiveHint=true; READ_ONLY tools lack readOnlyHint=true. This prevents LLMs from understanding operation safety without reading descriptions.
Parameter constraints under-specified: numeric parameters (bin_s, max_health_score, min_duration_s, top_k, count, max_samples) lack min/max bounds in descriptions. LLMs may pass absurd values (e.g. bin_s=999999, count=1000000) causing timeouts or memory exhaustion.
Error handling guidance absent: no tool describes what to do on failure (e.g. 'If indexing fails, check disk space and retry' or 'If topic not found, call inspect.list_topics first'). LLMs receive no recovery path on errors.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 61 | 2026-07-28+ | v2 |
Diff two missions signal by signal, ranked by effect size. `align` matters: "progress" normalises each mission to 0–1 of its own duration (the right default when runs differ in length), "absolute" compares raw seconds, and "event" anchors each mission on the first instant it actually moved — use that when runs idle for different lengths before starting, since otherwise every later difference is an artefact of the offset. The anchor instants are reported. Returns effect sizes and the most-changed list, never raw data.
A Foxglove layout JSON pre-pointed at the incident. baglens is not a visualiser — Foxglove is, and it is better. Hand off to the tool people already use, with the panels and the timestamp already set up.
A standalone interactive HTML plot for a set of signals. `signals` are "topic:field.path" strings, e.g. "/odom:twist.twist.linear.x". Opens in any browser with no server.
A Markdown investigation report where every claim carries its citation. Includes the verdict, the caveats, every finding with the rule that fired and the numbers behind it, and the per-topic health table. This is the artifact to paste into an issue.
Write a small MCAP containing only the interesting window and topics. This is how you hand the evidence to a colleague: a 40 MB slice instead of a 40 GB recording, containing exactly the failure. The source file is never modified.
A grid of frames across a window as a single labelled image. One picture instead of twenty: cheaper in context and easier for a vision model to compare across, because the frames sit side by side with their timestamps.
Camera frames near given timestamps, written as PNG files you can open. Give `times` explicitly, or `around_s` with `count` to sample either side of an incident. Prefer frames.contact_sheet when you want to *compare* frames — one image is far cheaper than several and easier to reason across.
Run a full integrity audit on a robot recording. ALWAYS call this before drawing conclusions from a bag — it tells you whether the data can support the analysis you are about to do. Returns findings ranked by severity, a per-topic health table, and explicit `caveats` describing what this recording cannot prove. If `verdict` is "unassessable", the tool is refusing to grade this recording rather than grading it badly: too few of its topics had a measurable rate, too little of the wall clock was covered, or it was too short to establish a baseline. `assessability.reasons` says which. In that case a short finding list means the checks did not run, NOT that the recording is clean — say so rather than reporting it as healthy. Start here, then use health.explain_finding to drill into a specific finding, health.find_gaps to see whether a silence was system-wide, and health.clock_report for the recorder-lag curve. `sensitivity` scales the gap and trend thresholds; use "high" on a quiet fleet and "low" on a noisy one. Safe on damaged and in-progress files.
List every silence, with the co-silent topics that explain it. This is the tool that answers "did the sensor die, or did the recorder stall?". Each gap carries a `concurrency` score and a `classification`: system_wide_stall (recorder, disk, CPU or power), subsystem_failure (a shared driver or bus — read `co_silent_topics`, that list is the diagnosis), or isolated_topic (that sensor or node alone). Gaps come back longest first. If the result is truncated, pass its `continuation_token` back to walk further down the list.
min/max/mean/std/percentiles for one numeric field. Cheap and high-value. `field_path` is dotted, e.g. "twist.twist.linear.x" or "ranges[0]". Prefer this over sampling messages when you only need the distribution.
The single message nearest a timestamp. The drill-down endpoint. Use after an anomaly or gap has been located, to see exactly what was published.
Topics, types, message counts and observed rates for one recording. Cheap — reads the index, not the messages. Call this before any query so you know what exists; call health.audit_recording before trusting what it says.
A few decoded messages, decimated across the range. For *shape*, not bulk. Hard-capped at 20. If you want values over time use timeseries.extract, which returns statistics instead of payloads and costs a fraction of the context.
The message definition for a topic, so you know which field paths exist. Call this before timeseries.extract or inspect.field_stats rather than guessing a field path and getting an empty series back.
Point counts, density and empty-return ratio over a lidar topic. Summarised statistically rather than rendered: a falling point count or a rising empty-return ratio is sensor degradation, and that shows up in the numbers long before it shows up in a picture.
Output schemas not exposed in tool definitions: Pydantic models (MissionList, IndexProgress, SourceAdded) exist in code but are not documented in the tool's input_schema or as a separate output_schema field. LLMs cannot predict response structure.
Pagination token handling unclear: catalog.list_missions and health.find_gaps accept continuation_token but descriptions do not explain how to use it or what happens if it expires. LLMs may retry with stale tokens.