Loading ToolBench
Canvas MCP server presents a moderate-quality implementation with 30 tools covering Canvas LMS features. Strengths: all tools have descriptions and input schemas with typed parameters. Weaknesses: descriptions lack depth and strategic guidance for LLM selection; output schemas are undocumented; error handling is absent; no parameter validation examples; tool composition could be tighter. The server implements basic JSON Schema structure but falls short of production-grade polish. Average tool score is 62/100.
Archive a conversation
Create a new conversation (send a message)
Delete a conversation permanently
Get download information for a file
Get grades for all enrolled courses
Get details about a specific assignment
Get details about a specific conversation
Output schemas are completely undocumented. No tool declares what fields it returns, their types, or relationships. LLMs cannot plan multi-step sequences without knowing what data comes back (e.g., does list_courses return course_id? course_name? both?). This forces agents into guesswork and requires 'try and see' exploration.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 59 | 2026-07-28+ | v2 |
| 2026-04-07 | F | 16 | - | v1 |
Get details about a specific course
Get grades for a specific course
Get information about the currently authenticated user
Get submission details for a quiz
Get submission details for an assignment
Get upcoming assignments and events
List announcements (optionally filtered by course)
List all assignments for a course
List calendar events within a date range
List all conversations (messages) in the user's inbox
List all modules for a course
List todo items for a specific course
List all courses for the current user
List files in a course or user's personal file storage
List quizzes for a course
List all todo items across all courses
Mark a conversation as read
Mark a conversation as unread
Reply to an existing conversation
Search for users in a course
Star a conversation for easy access
Submit an assignment
Remove star from a conversation
Descriptions are too generic and lack strategic context. Examples: 'Get information about the currently authenticated user' does not explain WHEN to call this vs. alternatives, what fields it returns, or whether it requires authentication checks. Baseline from rubric: descriptions should be 10 - 1024 characters and answer WHAT, WHEN, and prerequisites. Most Canvas descriptions are 40 - 70 characters and answer only WHAT.
No error handling or recovery guidance. If a call fails (e.g., 'course not found', 'submission invalid'), the LLM has no instruction on what to do next. Rubric requires error responses to say: 'User not found. Try search_users() with a partial name.' Current implementation likely returns raw API errors or generic 500 responses.
Parameter validation rules are missing. For example, list_conversations has an enum on 'scope' (inbox, unread, starred, sent, archived), which is good, but list_calendar_events accepts 'start_date' and 'end_date' as strings with no format constraint, range check, or example. LLMs will pass invalid dates (e.g., '2024-13-45'). Descriptions should state: 'ISO 8601 format only (YYYY-MM-DD), start_date must be before end_date'.
Destructive operations lack confirmation or dry-run. delete_conversation is marked DESTRUCTIVE but has no confirmation step, dry-run option, or warning in the description. Agents make mistakes, a single hallucinated call can delete a user's conversation. Rubric requires: 'Irreversible operations should support a dry-run or confirmation step.'
No pagination support visible in list tools. list_courses, list_assignments, list_conversations, list_files, list_quizzes, list_todos, and list_announcements have no limit, offset, or cursor parameters. If a user has 500+ courses, the response will bloat context. Rubric baseline: list tools should accept page/offset and limit (default ~20), and return a total count or next_cursor.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) visible. Tools like delete_conversation, submit_assignment, and create_conversation should declare destructiveHint: true so MCP clients can warn or confirm before execution. Tool annotations are part of the current 2026-07-28 spec.
Conversation management tools (mark_read, mark_unread, star, unstar, archive) are too fine-grained. Five separate tools to manipulate a single conversation state invite orchestration overhead and LLM confusion. Consider combining into a single 'update_conversation' tool with action enums, or batch operations.
Required parameters lack human-friendly identifiers. Many tools require numeric IDs (course_id, assignment_id, conversation_id) but users speak in names ('Math 101', 'Calculus Midterm', 'Chat with Professor'). Tools should accept both IDs and names, or the server should provide a search/lookup first. Current design forces extra discovery calls.
Missing context in multi-step workflows. For example, list_courses returns courses, but does it return course_id, course_name, and enrollment_state in the response? If get_course requires course_id, the list response MUST include course_id. Undocumented output schemas make tool chaining unclear.