Benchmark harness for measuring MCP (Model Context Protocol) performance across multiple database backends (PostgreSQL, MySQL, SQL Server, SQLite) via Faucet database abstraction layer
Tools have basic descriptions and parameter schemas, but lack depth and LLM-optimization. Descriptions are functional but generic (avg ~40 chars, below 194-char baseline). Parameter descriptions exist but are minimal. No output schemas documented. Error handling guidance absent. Tool names follow verb_noun convention (good), but descriptions don't explain WHEN to use each tool or dependencies between them. No enum constraints on string parameters like 'filter' and 'order'. Security-critical tools (faucet_delete, faucet_raw_sql) lack confirmation/dry-run patterns.
Delete records from a table matching a filter
Describe schema of a table in a database service
Insert records into a table
List available database services
List tables in a database service
Query a table with optional filtering, ordering, and limit
Execute raw SQL query against a database service
No output schemas documented. LLMs cannot predict what fields are returned or plan downstream calls. Affects all 8 tools.
String parameters (filter, order, sql) lack enum constraints or format descriptions. LLMs will hallucinate invalid SQL syntax or WHERE clauses.
Destructive tools (faucet_delete, faucet_raw_sql) lack dry-run or confirmation patterns. No recovery guidance if agent makes a mistake.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 60 | <=2025-11-25 | v2 |
Update records in a table matching a filter
Descriptions are too brief (avg 40 chars vs 194-char baseline). Do not explain WHEN to use each tool, dependencies, or prerequisites. E.g., faucet_query description lacks mention of pagination or result limits.
No error handling guidance. Tools do not document what errors are retryable, user-fixable, or fatal. No recovery suggestions (e.g., 'Try faucet_list_tables first').