Loading ToolBench
GitHat CLI — scaffold apps and manage the skills marketplace
This is a CLI tool wrapper (create-githat-app) that registers 8 tools via MCP. The implementation has moderate structural quality but significant gaps in LLM-optimization. Tool names are reasonably clear action-verbs (create, deploy, search, list, install, installed, init), but descriptions are inconsistent, some are concise and helpful, others are vague or missing context. Input schemas are present for most tools but lack granularity in type definitions and constraints. Parameters are described but many lack usage context and validation rules. Output schemas are entirely undocumented, the LLM has no visibility into what these tools return, making tool chaining and multi-step planning difficult. Error handling is minimal; no recovery guidance is visible. The tools themselves appear functional (file paths exist, code is present) but the *definitions* exposed to LLMs are incomplete.
Scaffold a new GitHat app
Deploy this app to GitHat (no AWS credentials required)
Manage GitHat skills marketplace
Initialize a new skill package
Install a skill to your project
List installed skills in current project
List available skills
Search skills by keyword
Output schemas completely undocumented for all 8 tools. LLMs have no visibility into what fields are returned, making tool chaining, multi-step planning, and error recovery impossible.
String parameters lack formal constraints (enums, regex, minLength/maxLength). E.g., 'type' param in skills search should declare enum [template, integration, ui, ai, workflow]; 'slug' should enforce lowercase-with-hyphens; 'limit' should have min=1, max=100. Descriptions mention these but JSON Schema doesn't enforce them, inviting LLM hallucination.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 46 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 49 | - | v1 |
No error handling guidance visible. The deploy command has robust error checks (readConfig validates appId/secretKey and throws descriptive errors), but the tool definition includes no error catalog or recovery hints. If deploy fails due to missing GITHAT_SECRET_KEY, the LLM sees only a raw error and doesn't know to suggest checking .env.local.
'skills' tool has no input schema visible and uses a noun name instead of verb, violates pattern:tool. Cannot verify if this is a real tool, a delegating container, or an alias. Per scoring rules, inferred tools cap at 50.
State-modifying tools lack explicit annotations. 'create', 'deploy', 'skills install', 'skills init' all write to disk but don't declare this in descriptions. LLMs need to know 'This tool CREATES a new project on disk' vs 'This reads metadata'. Per spec 2026-07-28, tools should use destructiveHint/idempotentHint annotations.
Mutually exclusive parameters not documented. E.g., 'create' has both 'ts' (use TypeScript) and 'js' (use JavaScript), logically exclusive, but descriptions don't say so. LLMs will pass both, causing undefined behavior.
Pagination missing. 'skills list' and 'skills search' return results but no offset/limit pagination guidance. If the skills marketplace has 1000+ items, returning all of them wastes tokens; LLM needs cursor-based or offset pagination to iterate.
Numeric parameter 'limit' (skills list) lacks min/max constraints. No guidance on default behavior if omitted. Best practice: declare limit=25 (default), min=1, max=100 in both schema and description.