Benchmark LLM models on MCP tools for askads (Yandex Direct / VK Ads / Yandex Metrica). Supports fixed (deterministic fixtures) and live (real MCP servers) modes.
Critical gaps across all dimensions. Tool descriptions are present but minimal (10-50 chars, well below 194-char baseline). No parameter descriptions visible in any tool schema, all 9 tools have empty or missing parameter documentation. Input schemas exist but lack type constraints and descriptions. No output schemas documented. Duplicate tool name 'get_statistics' (tools 6 and 9) creates ambiguity. Tools are read-only (good) but lack error handling guidance, recovery paths, and actionable error messages. No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Descriptions are in Russian, which may limit LLM understanding in English-first contexts. No evidence of pagination, result limits, or chaining IDs in responses.
Информация о рекламном аккаунте.
Справочник гео-регионов.
Статистика кампаний за период: показы, клики, расход, CTR, конверсии.
Веб-аналитика: визиты, пользователи, конверсии по целям.
Группы объявлений кампаний.
Список кампаний (Id, Name, Type, State, Status).
Счётчики Яндекс Метрики.
Duplicate tool name 'get_statistics' (tools 6 and 9), LLMs cannot distinguish between Yandex Direct campaign stats and Yandex Metrica web analytics. Rename one to 'get_metrica_statistics' or 'get_analytics_statistics'.
Zero parameter descriptions across all 9 tools. LLMs cannot infer what 'ids', 'campaignIds', 'reportType', 'dateRangeType', 'fieldNames', 'counterId', 'metrics' mean or how to populate them. Every parameter needs a description explaining its purpose, format, and constraints.
No output schemas documented. LLMs cannot plan downstream tool calls or extract required fields (e.g., campaign IDs from list_campaigns to pass to get_statistics). Document what each tool returns: field names, types, and which fields enable chaining.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 41 | 2026-07-28+ | v2 |
Цели/конверсии счётчика.
Ключевые слова кампаний/групп.
Descriptions are 10 - 50 characters (well below 194-char baseline). 'Информация о рекламном аккаунте' (32 chars) lacks context for when/why to call the tool. Expand to 50 - 200 chars: state WHAT it does, WHEN to use it, and what it returns.
No error handling guidance. Tools return no recovery hints (e.g., 'If campaign not found, call list_campaigns() first'). No categorization of errors as retryable, user-fixable, or fatal. LLMs cannot self-correct on failures.