Loading ToolBench
NTU COOL (Canvas LMS) query server — reads plus human-confirmed grade/announcement writes against the Canvas API deployed at cool.ntu.edu.tw
Strong tool naming (all verb-first, clear actions) and comprehensive descriptions (avg 180 chars, well above 10-char minimum). All 10 tools have input schemas with typed parameters. Critical strength: two-step write gates (preview→confirm) implement the confirmation-request pattern for irreversible operations. Weaknesses: parameter descriptions in complex tools (preview_grade_change, post_grades) lack format/constraint details; output schemas not documented; no pagination guidance for list tools; error handling guidance absent from descriptions.
List announcements posted in a course
List assignments in a course
List courses the token owner is enrolled in or teaching
List the enrollment roster for a course
List files uploaded to a course
List submission/grading status for every student in a course
STEP 2 of 2 for posting announcements. Requires a confirmation_token from preview_announcement, then asks the user directly through the client's own confirmation dialog. The write happens only if the user accepts.
Output schemas not documented. LLMs cannot infer what list_courses, list_assignments, etc. return (field names, types, structure). Forces agents to guess at response shape and risks failed downstream tool calls.
List tools lack pagination guidance. No mention of limit, offset, page_size, or result count in descriptions. Large course enrollments could blow context window without pagination.
preview_grade_change 'grades' parameter has type 'array' with items 'object' but no schema for the object structure. Description says '{user_id: int, grade: string}' but this is not enforced in JSON Schema, LLMs may pass malformed objects.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 64 | 2026-07-28+ | v2 |
STEP 2 of 2 for posting grades. Requires a confirmation_token from preview_grade_change, then asks the user directly through the client's own confirmation dialog. The write happens only if the user accepts.
STEP 1 of 2 for posting announcements. Renders exactly what would be posted and returns a confirmation_token bound to that exact payload. Performs no write. Show the returned preview to the user before calling post_announcement.
STEP 1 of 2 for posting grades. Renders what would change (current grade → new grade, per student) and returns a confirmation_token. Performs no write. Show the returned preview to the user before calling post_grades.
No error handling guidance in tool descriptions. If a course_id is invalid, what should the LLM do? Descriptions do not explain recovery paths (e.g., 'Call list_courses first to find valid IDs').
Token expiration (10 minutes) and single-use nature of confirmation_token mentioned in server instructions but not in post_grades/post_announcement parameter descriptions. LLMs may not understand why a token suddenly fails.