Loading ToolBench
MCP server for Matrix — interact with Matrix homeservers via Model Context Protocol. Send/read messages, manage rooms via matrix-nio
16 tools with complete input schemas and descriptions. Naming is verb-first and clear (send_message, read_messages, list_rooms). Descriptions are adequate (100-200 chars typical) but lack actionable context for LLM selection, e.g., send_message vs send_direct_message distinction is unclear. Parameters are well-typed with enums (format: text|html|markdown) and defaults. No output schemas documented, LLMs cannot predict response structure. Error handling maps common nio responses to friendly messages but lacks recovery guidance. Missing tool annotations (readOnlyHint, destructiveHint). Composition is sound: tools are single-responsibility and chainable (e.g., resolve_alias → send_message). Security: credentials injected via environment (MATRIX_HOMESERVER, MATRIX_USER, MATRIX_PASSWORD, MATRIX_ACCESS_TOKEN), no secrets in params. Baseline: 549 production tools average 18-char names (this server: 10-25 chars ✓), 194-char descriptions (this server: 100-200 chars ✓), 100% of A+ tools have docstrings (this server: 100% ✓). However, lack of output schema documentation and missing tool annotations prevent A-grade.
Create a new Matrix room.
List all direct message rooms (1-to-1 conversations).
Get messages from a Matrix room around a specific date/time.
Get unread notification and highlight counts for a room.
Get detailed information about a Matrix room — name, topic, members, aliases.
List joined members of a Matrix room with display names and avatars.
No output schemas documented. LLMs cannot predict response structure (fields, types, pagination). Forces agents to guess what data is available after each call.
Missing tool annotations (readOnlyHint, destructiveHint, idempotentHint). LLMs cannot distinguish safe reads from destructive writes without explicit hints.
Descriptions lack actionable context for LLM selection. send_message vs send_direct_message distinction unclear, when should LLM choose one over the other? No dependency hints (e.g., 'resolve_alias first if you only have a room name').
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 61 | <=2025-11-25 | v2 |
Get a user's profile information — display name and avatar.
Invite a user to a Matrix room.
Join a Matrix room by ID or alias.
Leave a Matrix room.
List all Matrix rooms the user has joined.
Read recent messages from a Matrix room. Returns messages in reverse chronological order.
Resolve a Matrix room alias to a room ID.
Send a direct message to a user. Creates a DM room if needed.
Send a message to a Matrix room. Supports plain text, HTML, and Markdown formats. Can reply to a specific message.
Send a reaction emoji to a message in a Matrix room.
Error responses map to friendly messages but lack recovery guidance. 'Failed to send message, check room permissions' tells LLM nothing about next steps. Should suggest: 'Try get_room_info to verify membership, or join_room if not a member.'
read_messages accepts 'since' pagination token but no documentation of token format or how to chain calls. LLMs cannot reliably paginate without explicit guidance.