Loading ToolBench
Research scanners measuring AI agent bounty markets, MCP ecosystem health, and x402 payment infrastructure. Includes GitHub bounty pattern detection, MCP registry analysis, and economic data collection.
All 7 tools have descriptions (good), but parameter schemas are severely underdeveloped. Most tools accept only 2-4 optional parameters with minimal type information. No input schemas visible in source code, only informal parameter lists in descriptions. Descriptions are verbose (200-400+ chars) but lack actionable constraints, enums, or format guidance. No output schemas documented. Error handling is absent. Tools are read-only (low risk) but lack the structure needed for reliable agent composition.
Measure a specific pattern in GitHub's public bounty-labelled issue space: freshly created, zero-star, non-fork repositories that carry the exact NAME of a well-known open source project, wrapped in a small paid bounty issue. Sweeps GitHub Search Issues with label:opire, collapses to distinct repositories, resolves via GraphQL API in batches, and flags decoy candidates based on stargazerCount <= 3, isFork false, and no parent.
Measure how a Frantic bounty claim actually ends: paid, rejected, or expired. Fetches the board listing from GET /v1/board, then fetches each bounty's claim_progress from its per-bounty endpoint. Reports claim progress counts (capacity, occupied, available, active, delivered, accepted, paid, revising, rejected, expired) separately for cash bounties and goodwill bounties.
Measure the GitHub side of the MCP server registry. Crawls registry.modelcontextprotocol.io/v0/servers (cursor-paged), collapses version records to one per server name, resolves each server's declared GitHub repository via GraphQL in batches, and queries repository metadata (nameWithOwner, isArchived, isFork, isPrivate, stargazerCount, forkCount, createdAt, pushedAt, updatedAt, diskUsage, licenseInfo, primaryLanguage, owner, defaultBranchRef, openIssues). Distinguishes PRESENT, ABSENT (NOT_FOUND), and UNKNOWN states.
Independently verify the ABSENT figure from mcp_github_scan.mjs using a polite random sample with control stratum. Samples both SUSPECT (repos called ABSENT) and CONTROL (repos called PRESENT) strata, probes each via anonymous HTTPS to github.com (no token, no GraphQL), and reports agreement percentages. Uses deterministic seeded PRNG (mulberry32) for reproducible sampling.
No input schemas visible in source code. Parameters documented only as informal descriptions (e.g., 'out': string, 'sample_prs': number). No JSON Schema with type, minLength, pattern, enum, or constraint declarations.
No output schemas documented. Tools return JSON results (repos.json, CSV, etc.) but LLMs cannot infer field names, types, or structure. Agents cannot plan downstream calls or extract specific fields reliably.
Descriptions lack actionable constraints. E.g., 'delay' param described as 'Milliseconds between per-bounty requests (optional, default 150)', no min/max bounds, no guidance on rate-limit implications. LLMs cannot validate or reason about safe values.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 37 | 2026-07-28+ | v2 |
Measure the MCP server ecosystem from public, keyless sources. Crawls registry.modelcontextprotocol.io/v0/servers (cursor-paged, no auth), collapses version records to latest-per-name, and enriches npm-published servers with real adoption numbers from api.npmjs.org/downloads/point/last-month/ (bulk queries, 100 per request). Deliberately skips PyPI due to rate limiting. Distinguishes NOT_FOUND from UNKNOWN outcomes.
Estimate how many npm packages the MCP registry names are genuinely absent using a polite random sample instead of a census. Samples two strata (suspect: packages the broken scan called missing; control: packages the broken scan resolved) and checks each via npm's download API at one request every 2.5 seconds. Uses deterministic seeded PRNG (mulberry32) for reproducible sampling. Computes Wilson score interval confidence bounds.
Measure the whole x402 economy from the public CDP Bazaar discovery API. Fetches all listings from https://api.cdp.coinbase.com/platform/v2/x402/discovery/resources (paginated, no auth). Filters to Base-mainnet USDC listings priced at <= $100, extracts lowest price across accepts entries, and computes estimated 30-day revenue as price × l30DaysTotalCalls. Categorizes listings by service type and reports aggregates by category.
No error handling or recovery guidance. Tools silently fail or return partial results (e.g., 'UNKNOWN' states in mcp_github_scan) with no actionable error messages. LLMs cannot distinguish retryable from fatal failures.
Tool names use '_scan' suffix (bounty_decoy_scan, mcp_github_scan) which is vague. Names do not clearly convey action (scan what? for what purpose?). Better: 'detect_bounty_decoys', 'audit_mcp_registry_repos', 'verify_npm_packages'.