Loading ToolBench
MCP server for Langfuse — wraps the Langfuse public REST API
14 tools with explicit schemas and descriptions, but significant quality gaps. All tools have descriptions (10-200 chars, mostly adequate), and input schemas are present with typed parameters. However, parameter descriptions are sparse or missing context. Output schemas are undocumented, LLMs cannot predict return structure. Error handling is minimal; no recovery guidance. Naming is verb-first and clear (list_, get_, create_, delete_), following conventions. No tool annotations (readOnlyHint, destructiveHint) despite clear risk levels. Composition is reasonable, tools are single-purpose and chainable. Security concern: database direct access (psycopg2) without visible input sanitization; API keys returned in plaintext from create_api_key without warning.
Create a new API key pair (pk/sk) for a project. Returns the secret key — store it immediately, it is not shown again.
Create a new project inside an organization.
Delete an API key by its ID.
Find a project by name or create it if it doesn't exist.
Get full details of a trace including observations.
List API keys for a project.
Output schemas undocumented. LLMs cannot predict return structure (e.g., what fields does list_traces return? Is there pagination metadata?). Forces agents to guess or fail on downstream tool chaining.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Tools are marked with Risk levels in metadata but not exposed to LLM via MCP annotations. Agents cannot distinguish safe reads from destructive writes without extra reasoning.
Pagination parameters (limit, page) present in list_* tools but no documentation of max limits, default values, or total count in response. list_traces accepts 'limit' but no guidance on max (100? 1000?). Agents may request excessive results.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 57 | 2026-07-28+ | v2 |
List all organizations.
List all local repo → Langfuse project mappings.
List all Langfuse projects accessible with the current API key.
List scores with optional filters.
List sessions.
List traces with optional filters.
Look up which Langfuse project a local repo/path is mapped to. Walks up the directory tree to find the nearest match.
Record that a local repo/path uses a specific Langfuse project.
create_api_key returns plaintext secret_key with no warning. Description says 'store it immediately, it is not shown again' but does not warn that this is a sensitive credential that should never be logged or echoed. Agents may inadvertently expose secrets.
No error handling guidance. Tools do not document what errors are possible, when they are retryable, or what the LLM should do next. E.g., delete_api_key raises ValueError if key not found, but no recovery hint.