Loading ToolBench
A web service and validation system for the FRED Protocol - a specification for making entities (organizations, products, APIs, etc.) discoverable by AI agents through a standardized JSON file
FRED Protocol exposes 4 tools via HTTP Next.js endpoints. All tools have descriptions and input schemas are partially visible. However, critical gaps undermine quality: (1) Tool names lack action verbs, 'claim', 'validate', 'get_fred_json', 'validate_fred' are weak; 'claim' is especially vague (does it register, create, or assert ownership?). (2) Parameter descriptions exist but are minimal (10-50 chars), below the 72-char baseline. (3) Output schemas are not documented, callers cannot predict response structure. (4) Error handling is generic ('Internal server error' at 500); no recovery guidance. (5) The 'claim' tool accepts 20+ optional fields with no dependency documentation, LLMs will struggle to know which combinations are valid. (6) No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite 'claim' being destructive (writes to blob storage). Baseline: 90% of A+ tools start with action verbs; 100% document return types; 94% have docstrings. This server has descriptions but they are too terse and lack LLM-optimization.
POST endpoint that accepts a ClaimBody with domain, name, type, description, url, email and optional protocol/capability/pricing/trust/context/discovery/policy fields. Builds a FRED JSON object, validates the domain format, and returns the generated fred.json along with a hosted URL
GET endpoint that retrieves a fred.json file for a given domain. Attempts to fetch from Vercel Blob storage first, then falls back to redirecting to the domain's /.well-known/fred.json
POST endpoint that accepts a FRED Protocol JSON object and validates it against the FRED Protocol specification v0.1. Returns validation result with level (0-3), errors array, warnings array, entity name, and entity type
Standalone validator function that validates FRED Protocol JSON files against the specification. Checks required fields, valid types, URLs, and compliance levels. Returns validation result with errors and warnings
Tool names lack action verbs. 'claim' is ambiguous (register? create? assert?). 'validate' and 'validate_fred' are duplicates. LLMs cannot infer intent from names alone.
Output schemas undocumented. 'claim' returns {fred_url, hosted_at, fred_json} but callers must infer structure. 'validate' returns {valid, level, errors, warnings, entityName, entityType} without formal schema. Baseline: 100% of A+ tools document return types.
Parameter descriptions are terse (10-50 chars vs 72-char baseline). E.g., 'domain' = 'Domain name (e.g., yoursite.com)' lacks context on format, validation, or when to use. LLMs need explicit constraints.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 44 | 2026-07-28+ | v2 |
'claim' tool has 20+ optional fields with no dependency documentation. LLMs cannot determine which combinations are valid (e.g., is protocol_type required if protocol_url is set?). Undocumented dependencies cause silent misuse.
No tool annotations. 'claim' is destructive (writes to blob storage, generates FRED JSON) but lacks destructiveHint. 'validate' and 'validate_fred' are read-only but lack readOnlyHint. Agents cannot distinguish safe from unsafe tools.