Loading ToolBench
The server defines 6 tools with mostly complete descriptions and reasonable parameter schemas. However, there are systematic gaps: (1) output schemas are not documented for any tool, LLMs cannot know what fields to expect in responses; (2) error handling and recovery guidance are absent, tools return raw API responses without actionable error messages; (3) parameter descriptions lack format constraints and validation guidance; (4) no tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite all tools being read-only or side-effect-free; (5) some tool descriptions are verbose and lack clear 'when to use' guidance. The naming is consistent (pbi_* prefix) and tools are well-decomposed (each has one responsibility), which is a strength. Compared to production baselines (avg tool description 194 chars, 100% of A+ tools with documented returns), this server falls short on output documentation and error guidance.
Authenticate to Power BI service. Checks for cached token first, attempts refresh if expired, falls back to device code flow if needed.
Aggregate dataset metadata + datasources + bound gateways + refresh schedule + impacted reports. Single call that returns everything needed to understand dataset configuration without making multiple round-trips. Useful for diagnostics and context-gathering.
One-shot diagnostic report for a Power BI dataset refresh failure. Pipeline: 1. Pull dataset metadata + datasources + gateways + schedule + impacted reports. 2. Pick a refresh to analyze: explicit refresh_id, OR latest Failed ViaEnhancedApi refresh. 3. Fetch refresh execution details (messages, objects, attempts). 4. Classify error: root cause table/partition, error code, underlying pattern, next actions. 5. Add PBIP source code locate hint (folder path, expressions.tmdl, table file).
Execute a DAX query against a Power BI dataset and return the result rows. Uses the Execute Queries REST API (POST executeQueries). Only DAX queries are supported; MDX and DMV are not.
Read local PBIP semantic model structure (TMDL files, no Desktop connection needed). Requires 'pbip_root' configured in config.json. Actions: - overview: Model summary - all tables with column/measure counts, relationship count. - table: Single table detail - columns, measures, partition M expression. Requires name. - measure: Single measure DAX expression + metadata. Requires name (fuzzy match across all tables). - expressions: Shared M expressions (datasource definitions). - relationships: All model relationships with from/to table.column and cardinality.
No output schemas documented for any tool. LLMs cannot plan downstream tool calls or extract required chaining IDs without knowing response structure.
No error recovery guidance. Tools lack actionable error messages. E.g., if device code flow times out, what should the agent do? If pbip_root is not configured, what is the error message? Raw API errors are not sufficient.
Missing tool annotations. All tools are read-only (marked risk=READ_ONLY), but there are no readOnlyHint annotations in the schema. This prevents clients from optimizing caching, retry logic, or agent planning. Per spec: tool annotations (readOnlyHint, destructiveHint, idempotentHint) are current patterns.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 58 | <=2025-11-25 | v2 |
Locate PBIP source code for a dataset (and optionally read a specific table's TMDL + M source). Requires 'pbip_root' configured in config.json (path to local PBIP repo root, e.g. 'C:/repos/my-project/data/power-bi-report').
Conditional parameter dependencies not formally enforced. pbi_local_model has action-dependent required parameters (name required for action=table/measure), but this is documented in description text only, not in schema constraints. LLMs may pass name when action=overview, wasting tokens.
pbi_local_model action parameter is not a formal enum. Constraint listed in description ('overview | table | measure | expressions | relationships') but not in JSON Schema as 'enum'. LLMs may hallucinate other actions.
Parameter descriptions lack format constraints and validation rules. Examples: workspace_id/dataset_id (UUIDs? strings? length?), query parameter (must start with EVALUATE? max length?), impersonated_user (email format? UPN format?).
Ambiguous tool selection. pbi_dataset_info and pbi_diagnose both pull dataset metadata. Description guidance on when to use each tool is missing. LLM may pick wrong tool, wasting a call.
Pagination and result limits not addressed. Tools returning lists or large results (pbi_local_model expressions, relationships; pbi_execute_query rows) do not document limits or pagination. Large responses will blow context windows.
External configuration (pbip_root) required but not gracefully handled in descriptions. What error is returned if pbip_root is missing? Should pbi_locate_pbip and pbi_local_model include setup instructions or a check tool?