Loading ToolBench
NyxDocs demonstrates solid tool definition quality with all 8 tools having clear verb-prefix naming, descriptions present for all parameters, and well-structured Pydantic schemas. However, there are notable gaps: (1) output schemas are not documented in the visible code, responses are returned as strings rather than structured objects, (2) error handling guidance is minimal, many tools return generic error messages without recovery hints, (3) descriptions, while present, are sometimes generic and lack actionable context (e.g., 'Returns project information' without explaining what fields to expect), (4) no tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite all being read-only operations. Parameter definitions are strong (all have types and descriptions), constraints are well-specified (enums for blockchain/category, numeric bounds), and naming follows verb_noun convention. All 8 tools are explicitly registered with schemas visible in crypto_tools.py and system_tools.py. The server shows good engineering fundamentals (mypy strict mode, test coverage config, proper async handling) but lacks the refinement needed for A-grade designation.
Check for recent documentation updates across projects or for a specific project. Returns list of recently updated documentation.
Retrieve actual documentation content for a cryptocurrency project. Returns the full text content of available documentation.
Get detailed information about a specific cryptocurrency project, including blockchain details and documentation status.
Get system statistics including total projects, documentation counts, and health status.
List all supported blockchain networks with project counts and information.
List all supported project categories with descriptions and project counts.
Output schemas not documented. Tools return string responses (e.g., 'str' type in search_crypto_projects async function) rather than structured objects with typed fields. LLMs cannot predict what fields to extract or what to pass to downstream tools.
No tool annotations. All tools are read-only operations (risk: READ_ONLY declared), but tool definitions lack readOnlyHint annotations. This prevents LLMs from reasoning about safety/reversibility without reading full descriptions.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 57 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 49 | - | v1 |
Search for cryptocurrency projects by name, blockchain, or category. Returns project information including available documentation.
Get help and examples for using NyxDocs search and query features.
Error handling lacks recovery guidance. Code shows inline error handling (e.g., 'Invalid blockchain: {params.blockchain}. Supported blockchains: [...]') but many failure paths are not visible. No explicit pattern for retryable vs. fatal errors; LLMs cannot distinguish whether to retry, ask user, or abort.
Description quality inconsistent and sometimes generic. 'get_system_stats' description 'Get system statistics including total projects, documentation counts, and health status' (92 chars) is acceptable but 'search_help' description 'Get help and examples for using NyxDocs search and query features' (65 chars) is vague, what exactly is returned? Is it documentation or interactive help? Tools like 'list_blockchains' and 'list_categories' lack context on when to call them vs. alternatives.
Pagination support unclear. 'search_crypto_projects' limits results to max 50, but output schema not visible, unclear if response includes total_count, next_cursor, or has_more for pagination. 'check_updates' limits to 100 results but no pagination context documented.
Parameter 'format' in get_documentation accepts free-form string ('markdown, html, text') without explicit enum constraint. LLMs may pass invalid values like 'json' or 'xml'. Should use enum constraint to self-document valid options.