Loading ToolBench
MCP server that exposes the Langfuse REST API as tools — query traces, observations, sessions, scores, prompts, datasets, and metrics from any MCP client.
langfuse-mcp demonstrates solid tool definition quality with consistent naming, comprehensive parameter schemas, and clear descriptions across 21 read-only tools. All tools follow verb_noun naming (list*, get*) and include input schemas with type definitions. Descriptions are present and actionable (avg ~120 chars). However, output schemas are not documented, responses are JSON-stringified without field-level documentation, forcing LLMs to infer structure. Error handling is minimal; no recovery guidance or actionable error messages visible. Parameter descriptions are good but lack some constraint details (e.g., 'orderBy' accepts free-form strings without examples). No tool annotations (readOnlyHint, idempotentHint) despite all tools being read-only and idempotent.
Fetch daily aggregated usage / cost / count metrics for traces and observations.
Fetch metadata for a dataset by its name.
Fetch a single dataset item by id.
Fetch a specific dataset run by name.
Run a metrics query (counts, latency, cost, token usage). Pass a JSON query string per the Langfuse metrics API.
Fetch a single observation by id.
Output schemas not documented. All tools return JSON-stringified responses without field-level documentation. LLMs cannot infer which fields are present, their types, or how to chain results to downstream tools.
No tool annotations despite all tools being read-only and idempotent. Missing readOnlyHint and idempotentHint annotations prevent clients from optimizing caching and retry logic.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 61 | 2026-07-28+ | v2 |
Fetch a prompt by name. Optionally pin to a specific version or label (e.g. 'production').
Fetch a single score by id.
Fetch a single score configuration by id.
Fetch a session by id, including its traces.
Fetch a single trace by id including all nested observations.
List items in a dataset (inputs / expected outputs / metadata).
List runs (evaluation rounds) for a dataset.
List datasets configured in Langfuse.
List models known to Langfuse (for cost / token attribution).
List observations (spans, generations, events) with filters.
List prompt definitions tracked in Langfuse.
List score configurations (definitions for score names, ranges, and categories).
List scores with filters.
List sessions within a time range.
List Langfuse traces with filters. Returns summary metadata (use getTrace for the full observation tree).
getMetrics parameter 'query' accepts free-form JSON string without schema or validation. LLMs cannot discover valid query structure; requires external documentation. Should expose query parameters as structured fields (view, dimensions, metrics, filters, timestamps).
No error handling or recovery guidance visible. Tools return raw API responses without actionable error messages. If a trace ID is invalid or not found, LLMs receive no guidance on what to try next.
Parameter 'orderBy' in listTraces lacks constraint documentation. Accepts free-form strings (e.g., 'timestamp.desc') without enum or pattern validation. Should document valid sort fields and order directions.