Loading ToolBench
A Python-based system for managing AI project rules, skill catalogs, and agent configurations with audit logging and confidence scoring
The AI Project Rules Generator server exhibits significant definition quality issues across nearly all dimensions. While it provides 22 tools with reasonable naming conventions (verb-prefixed), the implementation suffers from incomplete schemas, minimal parameter documentation, and lack of output schema specification. Most tools have descriptions but many lack the depth needed for LLM-driven selection. No explicit input schema validation is visible in the codebase for most tools, and error handling guidance is entirely absent. The server appears designed as a project utility rather than an agent-facing MCP tool, with minimal consideration for LLM interaction patterns.
Extract lightweight capability summaries from skill entries
Extract design tokens from Tailwind, CSS, and SCSS files
Score semantic similarity between a query and a set of tags
Search all discovered skill sources for keyword matches
Decorator for audit logging of actions with session tracking and error handling
Build a skill catalog from project root with optional incremental updates
Build a single unified catalog from multiple skill sources with source type and name tagging
No explicit input schemas visible in MCP tool registration. Parameters are documented in descriptions but not in formal JSON Schema with type constraints, enums, or validation rules.
Output schemas are not documented. Tools return various types (Path, tuple, dict, lists) but return value structure is not formally specified for LLM planning.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 46 | <=2025-11-25 | v2 |
| 2026-03-13 | F | 23 | - | v1 |
Compute a confidence score report for the project
Compute the difference between old and new state dictionaries
Analyze historical audit logs and warn about significant confidence score drops
Detect native MCP servers from project configuration files
Format a confidence report into a human-readable summary string
Generate a human+AI-readable Markdown skill map from the catalog, grouped by source type
Return a list of parsed audit logs from a directory
Load MCP registry from YAML file with server configurations
Load runtime default configuration from rules config file
Refresh project state memory by summarizing recent audit logs
Rotate audit logs in a directory, keeping only the max_logs most recent files
Route MCP servers based on user intent and tech stack matching against registry
Summarize recent audit logs into project state memory
Validate skill catalog for staleness and missing entries
Display audit logs from log directory in formatted table or detailed view
No error handling or recovery guidance. Tools have no documented error paths, retryability, or actionable error messages for LLM fallback planning.
Several parameter descriptions lack specificity. E.g., 'threshold' in compute_confidence_report lacks units/range; 'tags' in SemanticMatcher.score lacks expected format or constraints.
Tools accepting Path parameters should accept string paths and resolve them. Requiring Path type forces type conversion in agent logic rather than accepting natural string input.
No pagination or limit enforcement visible. Tools like list_audit_sessions could return unbounded results, risking context window exhaustion.
Composite tool names suggest multiple responsibilities. audit_logger is a decorator (logging concern); SkillDiscovery.search and CapabilityExtractor.extract mix discovery and extraction in ways that could be separate tools.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) visible. Tools with side effects (rotate_audit_logs, refresh_project_state) should be marked as destructive/non-idempotent.
Parameter 'sources' in build_unified_catalog is typed as 'list | None' with no type constraints on list elements. Should be 'list[SkillSourceConfig]' with documented structure.