Loading ToolBench
Local-first MCP server for Canvas LMS. Stdio transport, no third-party broker.
Canvas Local MCP has 10 tools with complete input schemas and descriptions, but several quality gaps reduce overall score. All tools are read-only with low risk. Naming follows verb_noun convention consistently (list_*, get_*, upcoming_*, planner_*, todo). Descriptions exist for all tools and parameters, but many are terse (16-80 chars, below the 50-200 char LLM-optimized baseline). Output schemas are not documented, the tool code returns raw API responses without declaring what fields agents should expect. Error handling is minimal: API errors are propagated without recovery guidance. Parameters lack constraints (enums, ranges, patterns) even where they should be present (e.g., course_id is just 'integer' with no range hint). The _get() helper function strips some API metadata (per_page defaults to 100, pagination handled internally), which is good, but responses still return verbose data (full course objects, submission details) without filtering for chat relevance. Tool composition is reasonable, each tool has a single responsibility, but some tools return large unfiltered lists (e.g., list_assignments with no limit parameter risks context window exhaustion). No tool annotations (readOnlyHint, etc.) are declared despite all tools being read-only. Security: credentials are injected via environment (.canvas.env), not exposed as parameters, good practice.
Get file metadata including download url.
Current grades. Per-course if course_id provided, otherwise all enrollments.
Fetch a course wiki page by its url slug.
Course announcements.
List assignments in a course. Includes due_at, points, submission status.
List enrolled courses. Returns id, name, course_code.
No output schemas documented. Tools return raw API responses without declaring what fields agents should expect. For example, list_courses returns course objects with id, name, course_code, but also includes additional fields like 'sis_course_id', 'workflow_state' not mentioned in descriptions. Agents cannot reliably extract required data or plan downstream calls.
Descriptions are terse and lack actionable context. Many descriptions are 16 - 80 chars (e.g., 'Course announcements.' is 22 chars, 'User's TODO list' is 16 chars). The LLM-optimized baseline is 50 - 200 chars. Short descriptions do not explain WHEN to use a tool vs. similar ones (e.g., when to call list_announcements vs. list_modules, or upcoming_events vs. planner_items). This forces LLMs to guess.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 64 | 2026-07-28+ | v2 |
Course modules with items.
Planner items for the user. ISO dates (YYYY-MM-DD).
User's TODO list (ungraded assignments to look at).
Upcoming planner items (assignments, calendar events) across all courses. Canvas returns roughly the next two weeks.
Input parameters lack constraints and validation hints. E.g., course_id is typed 'integer' with no range or validation hint. Which course IDs are valid? Are there system limits? LLMs may pass invalid IDs without guidance. Similarly, date parameters (start_date, end_date in planner_items) are 'string' with no format enforcement, no regex pattern, no enum, no constraint in the description beyond 'ISO format (YYYY-MM-DD)'.
No pagination controls in tool definitions. list_assignments, list_announcements, list_modules, and similar tools return paginated Canvas API results, but the tools expose no limit or offset parameters. The _get() helper internally handles pagination and returns all results (up to per_page=100 per request, looping through next links). Returning unlimited results risks context window exhaustion. No tool documents result limits or offers pagination parameters.
No error recovery guidance. If _get() raises an httpx exception (e.g., 404 for invalid course_id, 401 for expired token, 500 from Canvas), the exception is propagated as-is. Agents receive a bare HTTP status or stack trace with no guidance on how to recover. For example, 'Course not found' should suggest: 'Use list_courses() to find a valid course_id.' A 401 should suggest: 'Check CANVAS_TOKEN in ~/.canvas.env.'
No tool annotations (readOnlyHint, idempotentHint). All 10 tools are read-only, but this is not declared in the tool definition. MCP spec supports toolAnnotations.readOnlyHint to signal safe-to-retry and safe-to-parallelize tools. Absence of this hint may cause agents to be overly conservative (e.g., retrying sequentially instead of in parallel).
Parameter descriptions missing for optional parameters. E.g., list_courses() has active_only: bool with description 'Filter to active enrollments only' (good), but list_assignments' include_submissions: bool is described only in the schema 'Include submission data in response', this is minimal. Descriptions like 'Defaults to false' or 'Set to true to fetch submission metadata (score, workflow_state)' would be more actionable.
Redundant or overlapping tool purposes. upcoming_events() and planner_items() both return user planner items. The difference is unclear from descriptions alone. upcoming_events: 'Upcoming planner items (assignments, calendar events) across all courses. Canvas returns roughly the next two weeks.' vs. planner_items: 'Planner items for the user. ISO dates (YYYY-MM-DD).' When should an agent use one vs. the other? The descriptions do not clarify the semantic difference (time range, content type, filtering options).