Loading ToolBench
Extract tacit team knowledge from GitHub PRs and CI failures → generate CLAUDE.md and .claude/rules/ files
Tacit provides 7 tools with complete input schemas and descriptions. All tools are read-only and well-scoped. However, descriptions are generic and lack LLM-optimized guidance on WHEN to use each tool vs alternatives. Parameter descriptions are present but minimal (avg ~40 chars vs baseline 72). No output schemas documented. Error handling is basic (connection errors, missing params) but lacks recovery guidance. Tool names follow verb_noun convention well (db_connect, github_fetch_*). No tool annotations (readOnlyHint, idempotentHint) despite all being read-only. Missing dependency hints (e.g., 'Call db_connect before db_inspect_schema'). Database tools require connection_string + db_type on every call, forcing repetition instead of leveraging the connection cache design.
Connect to a database (PostgreSQL or SQLite) in read-only mode. Stores the connection for reuse by other db_ tools.
Inspect the full database schema: tables, columns, types, constraints (primary keys, foreign keys, unique, check, not null). Returns structured JSON.
Execute a read-only SQL query against a connected database. Returns results as JSON.
Fetch a sample of 10 rows from a database table. Useful for understanding data patterns, formats, and domain terminology.
Fetch all comments and review comments for a specific pull request. Includes both issue-style comments and inline code review comments.
Fetch pull requests from a GitHub repository. Returns PR metadata including title, number, author, labels, comment counts, review states, and first-timer flags.
No output schemas documented. LLMs cannot predict response structure, forcing them to guess at field names for downstream tool calls or data extraction. Baseline: 100% of A+ tools document return types.
Database tools require connection_string + db_type on every call despite db_connect caching connections. This forces repetition and wastes tokens. Design should accept optional connection_string (use cached) or require it only on first call.
Descriptions lack LLM-optimized guidance on WHEN to use each tool. E.g., db_sample_data says 'Fetch 10 rows' but doesn't explain when to call it vs db_query_readonly. Baseline: descriptions should answer WHAT, WHEN, and WHAT IT RETURNS.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 64 | 2026-07-28+ | v2 |
Fetch repo tree, recent commits, and branch rulesets in ~6 API calls. Returns file paths (naming conventions, test layout, tooling), commit message patterns, and enforced branch policies.
No tool annotations despite all tools being read-only. Missing readOnlyHint would help agents understand safety and retry behavior. Baseline: A+ tools use tool annotations for safety classification.
Error handling is minimal. db_query_readonly returns raw SQL errors without guidance. Should categorize as retryable (connection timeout) vs user-fixable (invalid SQL) and suggest recovery steps.