Loading ToolBench
MCP server that exposes Microsoft 365 privacy/compliance audit checks as tools for Claude Desktop / Cursor / any MCP client.
Server has 5 well-intentioned audit tools with clear domain focus (M365 compliance). All tools have descriptions (194 - 280 chars, within baseline 34 - 392 range) and input schemas with type definitions. However, output schemas are undocumented, LLMs cannot predict response structure for downstream chaining. Parameter descriptions are present but generic (e.g., 'Optional location filter' lacks format/enum guidance). No error handling guidance, no recovery paths, and no tool annotations (readOnlyHint, idempotentHint). Naming is verb-forward and clear (check_, find_, audit_, list_, summarize_), but composition could be tighter, find_orphaned_documents conflates two criteria (no owner OR stale) without separate tools.
Audit Conditional Access policies for risks. Lists every CA policy and flags any that aren't in `enabled` state (i.e. `reportOnly` or `disabled`). Report-only policies are documented intent that isn't actually enforcing; long-lived report-only policies are a common audit finding.
Return the tenant's M365 privacy / compliance configuration with verification status. Mirrors the structure of the m365-privacy-config checklist (data residency, Copilot no-training, Azure OpenAI retention, BAA scope) so the output can be pasted straight into a sign-off document for the compliance owner. Returns a dict with the tenant metadata, a summary count of verified vs outstanding items, and per-item evidence (or `action_required` if not yet verified).
Find documents in the tenant that look orphaned. A document is flagged if EITHER: - it has no owner (when include_no_owner is True), OR - it hasn't been accessed in `days_threshold` days. Each flagged document includes a recommendation (archive, reassign, or review). Useful for monthly governance review or before enabling Copilot on a site (orphaned docs leak into Copilot grounding by default).
List Data Loss Prevention (DLP) policies. Optional `location` filter (e.g. "Exchange", "SharePoint", "OneDrive", "Teams"). Returns the policy id, displayName, state, locations, and sensitiveTypes covered. Useful for confirming that the surfaces a Copilot or AI build will touch are actually covered before go-live.
Output schemas are not documented. LLMs cannot predict response structure (fields, types, nesting) for downstream tool chaining or data extraction. E.g., check_tenant_privacy_config returns a dict with 'tenant', 'summary', 'items' keys, but this is invisible to the agent.
No error handling guidance. Tools return dicts with 'error' keys (e.g., summarize_copilot_usage returns {"error": "Unknown team_id"}) but do not guide LLM recovery. Missing pattern: 'Try search_teams() first' or 'Available team_ids: [...]'.
Parameter descriptions lack format/constraint guidance. 'Optional location filter (e.g. Exchange, SharePoint, OneDrive, Teams)' uses examples instead of enums. LLMs may hallucinate invalid values like 'Slack' or 'Gmail'. Should declare: location enum: ["Exchange", "SharePoint", "OneDrive", "Teams"].
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 54 | 2026-07-28+ | v2 |
Summarize Microsoft 365 Copilot usage. With no arguments, returns a tenant-wide summary plus per-team breakdown sorted by adoption percentage (lowest first - the rollout-gap teams). With `team_id`, returns the detailed stats + top prompts for that team. Adoption is `activeUsers / totalUsers`. Teams below 50% get `adoptionStatus: "low"` so the user can see where rollout has stalled.
find_orphaned_documents conflates two independent criteria (no owner OR stale) into one tool. Composition pattern suggests splitting into separate tools (find_documents_without_owner, find_stale_documents) so agents can compose them independently.
No tool annotations (readOnlyHint, idempotentHint, destructiveHint). All tools are read-only audit operations, but this is not declared. Agents cannot infer safety properties without explicit hints.