Loading ToolBench
Safety-first Skyrim mod engineering, automation fabric, and Nexus publication gate
Skyrim Forge exposes 38 tools with complete JSON schemas and basic descriptions. However, parameter descriptions are consistently empty or minimal (most lack any explanation of what they control), violating the pattern:tool-description requirement that every parameter needs a description. Tool descriptions themselves are present but terse (avg ~40 chars, well below the 194-char baseline for A+ tools). No output schemas are documented. Error handling is absent from tool definitions, no guidance on recovery, retryability, or actionable error messages. Tool naming follows verb_noun convention (forge_*) but lacks clarity on distinctions between similar tools (e.g., forge_tool_import vs forge_tool_import_all, forge_tool_configure). No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear risk classifications in the metadata. Security-critical tools like forge_nexus_publish expose API keys as parameters, violating pattern:secret-injection.
Inspect ZIP/7z/RAR contents safely.
Inspect BSA/BA2 through the configured hash-pinned BSArch.
Create a Skyrim SE/AE BSA from a Forge workspace directory and reopen it with BSArch.
Extract BSA/BA2 through BSArch into Forge workspace staging.
Return the evidence-level capability registry or one capability.
Decompile PEX files using a configured hash-pinned Champollion into workspace staging.
Change one allowlisted Forge configuration value.
Read sanitized Forge configuration.
Parameter descriptions are empty or missing across all 38 tools. The rubric requires every parameter to have a non-empty description explaining what it controls. Examples: forge_config_set has 'key', 'value', 'approved' with empty descriptions; forge_tool_import has 'source', 'identifier', 'sha256', 'approved' with empty descriptions. This violates pattern:tool-description and forces LLMs to infer parameter meaning from names alone, increasing errors.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 46 | 2026-07-28+ | v2 |
Run the configured hash-pinned DeadMesh dmscan and capture reports in the workspace.
Find known Skyrim tool executables without changing configuration.
Inspect Forge configuration and automation readiness.
Build a FOMOD installer from a validated source tree into Forge staging.
Validate FOMOD XML, file coverage, branch metadata, flags, source paths, and destination paths.
Build a source-locked framework configuration transactionally and lint the result.
Validate a typed framework configuration plan.
Lint source-locked SPID, KID, BOS, SkyPatcher, FLM, and CDF/CID profiles without rewriting unknown syntax.
Inspect a loose mod directory.
Statically audit a CommonLibSSE-NG project and its lifecycle/runtime declarations.
Inspect a Windows x64 SKSE DLL PE structure and imports.
Build an audited native project with pinned CMake/vcpkg tools into Forge staging.
Validate a source-locked CommonLibSSE-NG native plugin plan.
Generate a bounded CommonLibSSE-NG project inside the Forge workspace.
Publish a mod to Nexus Mods using a Nexus API key and mod metadata.
Analyze Papyrus identity, inheritance, import conflicts, and optimization heuristics without rewriting source.
Compile Papyrus through a hash-pinned Bethesda compiler into Forge staging, verify fresh PEX files, and emit a build manifest.
Inspect TES4 header and masters. Header evidence only.
Read plugins.txt.
Query plugin records by signature, EditorID, or raw FormID.
Run all built-in deterministic Forge regression fixtures.
Run the dedicated Synthesis.Bethesda.CLI pipeline command. The GUI executable is not accepted.
Configure and SHA-256 pin an existing legal tool installation without copying it.
Transactionally import a recognized local tool into the private Forge tool vault and SHA-256 pin it.
Import all unambiguous recognized automation tools from one local source; GUI tools are opt-in.
Resolve the best configured hash-pinned real tool for one exact capability.
Recursively inspect a tool directory or ZIP without launching executables.
Inspect one configured tool, including pinned hash status.
Report the private tool vault and all configured catalog tools.
Return Skyrim Forge version.
Tool descriptions are terse (avg ~40 chars, well below 194-char baseline). Examples: 'Return Skyrim Forge version.' (30 chars), 'Inspect Forge configuration and automation readiness.' (54 chars), 'Run all built-in deterministic Forge regression fixtures.' (57 chars). Descriptions lack context on WHEN to use the tool, WHAT it returns, and any prerequisites. This violates pattern:tool-description and reduces LLM selection accuracy.
forge_nexus_publish exposes 'nexus_api_key' as a required parameter. This violates pattern:secret-injection, credentials must never appear as tool parameters. Secrets in parameters are logged in traces and prompt history, creating a security leak. Use server-side secret injection via environment variables or a vault instead.
No output schemas are documented for any tool. The rubric requires documenting the output schema so LLMs know what fields to expect and can plan downstream tool calls. Without documented outputs, LLMs cannot reliably chain tools or extract required data for follow-up operations.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) are present despite clear risk classifications in metadata (READ_ONLY vs WRITE). The spec requires tools to declare their safety properties so agents can reason about side effects and retry safety. This is a current pattern (2026-07-28 spec) and should be implemented.