Loading ToolBench
Comprehensive analysis and monitoring system for Model Context Protocol servers across multiple data sources, including data collection, classification, and analysis pipelines
This MCP server exposes 21 tools that appear to be primarily Makefile targets and internal Python script functions, NOT properly registered MCP tools with schemas and descriptions. Evidence: (1) Tool definitions come from Makefile comments and scattered Python scripts with no visible MCP tool registration code. (2) Only 2 of 21 tools have explicit input schema documentation (data_task_clusters with k2 integer; enrich_with_metadata and enrich_servers_with_tools with string paths). (3) Descriptions vary from vague ('Collects MCP server data...') to moderately detailed, but lack LLM-actionable guidance on WHEN to use each tool, WHAT happens to state, or what errors to expect. (4) No parameter descriptions for any tools except the 3 that have visible schemas. (5) No output schemas documented for any tool. (6) Error handling, recovery guidance, and permission checks are absent. (7) Tool names follow no consistent verb-noun pattern and several contain 'and' (e.g., 'data_initial_clean', 'data-cl-all'), suggesting bundled responsibilities. (8) No evidence of tool annotations (readOnlyHint, destructiveHint, idempotentHint), pagination support, or proper composition. This server appears to be a research/analytics orchestration tool, not a production MCP server. The Makefile is a task runner, not an MCP implementation.
Aggregates tool-level classifications by server_id, computing highest automation functionality, most common sub-functionality, and O*NET task levels
Calculates Cohen's Kappa for pairwise agreement between human and LLM ratings
Creates Inspect AI samples from tool data with formatted text input and metadata preservation
Detects AI-created servers via git history mining and validates results with pytest
CLServers pipeline for server classification including finance identification and NAICS industry classification
Enriches CLServers with aggregated tool classifications from CLTools
CLTools pipeline for O*NET task mapping and tool classification, with data matching and aggregation
No visible MCP tool registration: Tools appear to be Makefile targets and Python function names, not formally registered MCP tools. No evidence of tool descriptor objects with names, descriptions, and input_schema properties.
Missing input schemas for 19 of 21 tools. Only data_task_clusters, enrich_with_metadata, enrich_servers_with_tools have documented parameter schemas. Remaining tools have no visible schema definitions.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 30 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 22 | - | v1 |
Filters README content using LLM (Claude Sonnet) with configurable temperature and connection limits
Collects MCP server data from all 4 sources: Smithery, GitHub, Official List, and Awesome List
Collects usage statistics including npm registry searches and download statistics from npm API
Creates unified datasets by processing and unifying data from multiple sources
Runs O*NET task clustering with configurable k parameter and performs task embedding matching
Enriches CLServers CSV with aggregated tool-level classifications from CLTools, computing highest_automation_func, main_automation_subfunc, main_onet_task_level1/2/3, and occupation_title_main
Enriches task output CSV with creation_date, use_count from CLServers, usage data from JSON, and occupation_title from O*NET task statements
Flattens tool records with server context, optionally filtering for finance-only servers
Builds 3-level O*NET hierarchy from CSV rows (L1 clusters, L2 clusters, L3 tasks)
Loads filtered MCP server dataset from JSON file for tool extraction
Loads human validation CSV file for tools from Gorilla experiments
Loads LLM tool classifications from CLTools CSV file for comparison with human ratings
Maps human responses to LLM scale for comparison, supporting categorical and ordinal mappings
Transforms human validation data from long form to wide format (one row per server per participant)
Parameter descriptions missing for most tools. Tools with schemas (enrich_with_metadata, enrich_servers_with_tools) have parameter descriptions, but 16+ other tools lack any parameter-level documentation.
No output schemas documented. No tool returns documented field types, making it impossible for LLMs to plan downstream calls or extract structured data.
Tool names lack action verbs or use vague verbs. Examples: 'data_initial' (does it initialize data? fetch it? transform it?), 'data_cl_servers' (cl is acronym-heavy and unclear). Names should follow verb_noun pattern (e.g., load_filtered_dataset is better).
No error handling or recovery guidance. No tools document what errors they may raise, whether errors are retryable, or what the LLM should do next on failure.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Risk field present for some tools but no formal annotations in tool descriptors.
Tool names bundle responsibilities ('data_initial_clean', 'data_cl_all'). These suggest compound operations that should be split into individual tools so LLMs can compose them independently.
No state management or idempotency documentation. Several tools write data (data_initial, data_clean_readmes, etc.); no guidance on idempotency, side effects, or whether re-runs are safe.
No pagination or result-limiting guidance. Data-intensive tools (data_collect_servers, data_collect_usage, extract_tools_from_servers) provide no limits or pagination, risking context window exhaustion.