Loading ToolBench
A queryable source of engineering constraints exposed over MCP. Aggregates constraints from multiple sources and exposes them via MCP tools for validation and discovery.
Three tools with clear, detailed descriptions (194 - 250 chars each) and explicit input schemas. Tool names follow verb_noun pattern (get_*, describe_*, validate). However, input schemas lack formal type definitions and constraints, parameters are documented in descriptions but not enforced via JSON Schema enums or patterns. Output schemas are described in text but not formally documented. Error handling is partial: get_constraints fails open gracefully, but validate catches ValidationUnavailable; no guidance on retryability or user-fixable errors. No tool annotations (readOnlyHint, idempotentHint). Composition is sound: three single-responsibility tools with clear dependencies (describe_scope → get_constraints → validate). Naming is unambiguous and action-oriented.
Discover the selector vocabulary present in the registry so you can build a correct scope instead of guessing. Output lists the distinct providers, resource_types (Terraform resource identifiers, e.g. 'aws_s3_bucket' not 's3_bucket'), environments, repos (tags like 'tag:data-plane'), categories, severities, sources, and relationship layers/interactions. Call this first if unsure of valid scope values.
Return engineering constraints relevant to a scope. Inputs: scope (providers, resource_types, environments, repos, relationship), optional version (bundle id; defaults to latest). resource_types use Terraform resource identifiers, e.g. 'aws_s3_bucket' (NOT 's3_bucket'); repos are tags like 'tag:data-plane'. If unsure of valid values, call describe_scope first, or simply omit a dimension (omitted dimensions are 'don't care' and broaden the match rather than excluding). Output: {available, bundle_id, constraints[]}. Fails open: on any error returns available=false with an empty constraints list so you can proceed.
Validate a candidate artifact against in-scope constraints by delegating to enforcement engines. Inputs: artifact (object), scope, optional version. Output: {bundle_id, passed, results[]} where each result has constraint, severity, kind, verdict, violations, guidance.
Input schemas lack formal constraints. Parameters documented in descriptions (e.g., 'Terraform resource identifiers, e.g. aws_s3_bucket') but not enforced via JSON Schema enums, patterns, or type definitions. LLMs cannot parse these constraints from text alone.
Output schemas not formally documented. Descriptions state output structure (e.g., '{available, bundle_id, constraints[]}') but no JSON Schema definition. LLMs cannot reliably extract fields or plan downstream calls without formal output types.
No tool annotations. Tools lack readOnlyHint, idempotentHint, or destructiveHint. Agents cannot determine which tools are safe to retry or which modify state without reading descriptions.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 57 | 2026-07-28+ | v2 |
Error handling incomplete. get_constraints fails open (good), but validate catches ValidationUnavailable and returns error dict without guidance on retryability or next steps. No classification of errors as retryable vs. user-fixable.
Parameter descriptions include example values ('aws_s3_bucket', 'tag:data-plane'). LLMs may reuse these literally in real calls instead of adapting to context. Replace with formal enums or patterns.