Loading ToolBench
A detective for your data. Zero-config data quality monitoring exposed as MCP tools for AI coding agents. Profiles warehouses and detects anomalies with structured results.
Scherlok MCP has 6 well-designed tools with clear verb-noun naming and comprehensive descriptions (avg 190 chars, within baseline of 194). All tools declare input schemas with proper JSON Schema types and parameter descriptions. Parameter constraints are declared (e.g., days as integer with default, fail_on as enum). Descriptions answer WHAT, WHEN, and PREREQUISITES. However, output schemas are not formally documented in the visible code, they are mentioned only in descriptions ('returns the anomalies found', 'returns the connection target') without explicit JSON Schema definitions. All tools are read-only, addressing security concerns by not exposing credentials. Error handling is implicit (ConnectionNotConfiguredError, ValueError for missing tables) but lacks actionable recovery guidance in the schema layer. This is a solid B-/B server: definitions are above average, but output documentation and error guidance prevent it from reaching the 75+ range.
Run a watch over all tables and return a CI-style pass/fail gate. `fail_on` is `"critical"` (default — fail only on CRITICAL) or `"warning"` (stricter — fail on WARNING or worse). Mirrors `scherlok ci`. Returns `passed` plus the severity counts the decision was based on.
Return anomalies recorded in the last `days` days (capped). Reads from the local profile store — does not re-profile the warehouse.
Profile tables and store them as the baseline for future anomaly checks. Pass `tables` to limit to specific tables, or omit to profile everything visible. The first profile of a table establishes its baseline; no anomalies are reported here. Run `watch` later to detect drift.
List the tables visible to the configured warehouse connection. Returns the table names (capped) and a total count. Use this to discover what can be profiled before calling `investigate` or `watch`.
Report the current monitoring state. Returns the connection target (password redacted), how many tables the connection can see, and how many anomalies were recorded in the last 30 days. A quick health glance without profiling anything.
Output schemas not formally documented. Descriptions mention return types ('returns the anomalies found', 'returns the connection target') but no explicit JSON Schema definitions are visible for tool responses. LLMs cannot reliably plan downstream operations or extract specific fields without structured output documentation.
Error handling lacks actionable recovery guidance. ConnectionNotConfiguredError and ValueError are raised, but the error messages (visible in code) lack suggestions for next steps. The ConnectionNotConfiguredError does provide a hint ('Set the SCHERLOK_CONNECTION environment variable'), but ValueError for missing tables ('Tables not found on this connection') does not suggest discovery alternatives.
No output field naming consistency documented. The schema shows 'tables' parameter as array of strings, but there is no visible JSON schema for the response structure (e.g., does list_tables return {"tables": [...], "total": N}?). Response field names must be documented so downstream tools can chain correctly.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 69 | 2026-07-28+ | v2 |
Profile tables and detect anomalies against the stored baseline. Returns the anomalies found (type, severity, message per table), capped. Pass `tables` to limit scope, or omit to watch everything. Tables with no prior baseline are profiled silently (their baseline is set for next time).
Results are capped (MAX_TABLES=1000, MAX_ANOMALIES=200) but caps are not declared in tool descriptions. Schema descriptions should state 'Returns up to 200 anomalies' to set LLM expectations and explain why pagination/filtering might be needed.
Tool annotations (readOnlyHint, idempotentHint) not visible in the schema definitions. All tools are read-only and idempotent (profiling and baseline checks are side-effect free in terms of data modification), but these hints are not declared to help the LLM reason about retry safety.