Loading ToolBench
Coverage continuity infrastructure — AI copilot for FQHC caseworkers determining Medicaid eligibility
MediAssist AI exhibits significant definition quality gaps. While 19 tools are registered with basic descriptions, most descriptions are generic and under 100 characters, failing to explain WHEN to use each tool or what distinguishes similar operations. Parameter descriptions are sparse or missing entirely. No input schemas are visible in the provided code, tool definitions appear inferred from FastAPI route signatures rather than explicitly registered with JSON Schema. Output schemas are undocumented. Error handling is minimal; most endpoints return generic HTTP exceptions without recovery guidance. The server conflates multiple concerns (eligibility checking, patient CRUD, session management, reporting) without clear composition patterns. Critical security issue: session_id is a user-provided string parameter with no validation, enabling session hijacking.
Run an eligibility check through the AI agent
Run an eligibility check with streaming response
Run an eligibility check for a specific patient by ID
LLM classifies document type
Add a new patient to the database
LLM extracts structured fields from document
Deterministic alert generation based on state + timing
No explicit JSON Schema definitions visible for tool inputs. Tool definitions appear inferred from FastAPI route signatures rather than registered with proper MCP tool schemas. This violates the tool registration pattern and makes it impossible for clients to validate inputs before calling.
Output schemas are undocumented. Tools return HTTP responses (JSON objects) but the structure is not declared. LLMs cannot plan downstream tool calls or extract required fields without knowing what fields to expect. Example: list_patients returns {patients: [...]}, but this is not documented.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 39 | <=2025-11-25 | v2 |
Return metrics from the most recent query (guardrail, QA, latency, etc.)
Get a specific patient by ID
List saved conversation sessions for a patient
Aggregate stats: by tier, by state, completion rate
Get a specific determination report
Retrieve conversation history for a session (in-memory or Postgres)
Health check endpoint returning server status and MCP sessions
List all patients from the PostgreSQL database
List all saved determination reports
List all active session IDs
Record caseworker override decision
Deterministic validation: dates within range, names match, required fields present
Session management lacks security controls. session_id is a user-provided string parameter with no validation, enabling session hijacking or cross-session data leakage. No permission checks verify that a user can access a given session. Agents can call get_session with any session_id and retrieve arbitrary conversation history.
Descriptions are generic and under 100 characters for most tools. They do not explain WHEN to use each tool, what distinguishes similar operations (e.g., check_eligibility vs check_patient_by_id), or what the tool returns. Example: 'Run an eligibility check through the AI agent' does not explain the difference from check_eligibility_stream or when to use each.
Error handling is minimal. Most endpoints return generic HTTPException with status codes and brief messages. No recovery guidance is provided. Example: 'Patient not found' does not suggest calling list_patients() to discover valid IDs. Agents cannot self-correct from errors.