Loading ToolBench

Reading the current corpus…

awaithumans-human-in-the-loop-ai-agents MCP server · F · ToolBench

awaithumans-human-in-the-loop-ai-agents

github.com/awaithumans/awaithumans-human-in-the-loop-ai-agents

Human-in-the-loop AI agents platform with self-hosted HITL review dashboard, FastAPI server, and support for multiple workflow orchestration frameworks (LangGraph, Temporal)

2026-07-28+STREAMABLE_HTTP2 tools

39/100

Definition Quality45

Protocol Readiness0

Supportability55

Spec posture

Inferred effective spec 2026-07-28+·current baseline 2026-07-28

Static source inference · medium confidence · evidence: Streamable HTTP

Current-spec patterns detected

Summary

This server exposes only 2 tools via HTTP/FastAPI. Both tools have descriptions present, but they are HTTP route descriptions (GET /api/setup/status, POST /api/setup/operator) rather than LLM-optimized tool descriptions. The setup_status tool has an empty input schema ({}), which is a critical gap. The create_first_operator tool has a reasonable input schema with 4 parameters, all with type and description, but the descriptions are minimal (1-2 sentences). Neither tool has documented output schemas. The server is not an MCP server, it is a plain FastAPI HTTP service. No MCP protocol implementation is visible in the provided code (no tool registration via MCP, no MCP transport, no MCP capabilities). This is a fundamental mismatch: the evaluation assumes an MCP server, but the codebase is a standalone REST API.

Tools (2)

create_first_operatorwriteauthsource verified57/100

POST /api/setup/operator - Create the first operator and issue their session in one shot. The dashboard posts here after the operator pastes the setup token and their desired credentials. On success, the response sets the session cookie so the operator lands directly on the queue.

setup_statusread onlysource verified32/100

GET /api/setup/status - Check if setup is needed and if bootstrap token is active

Issues (5)

CRITICALtool ↗

No MCP protocol implementation detected. Codebase is a plain FastAPI HTTP service, not an MCP server. No tool registration, MCP transport, or protocol handshake visible.

setup_statuscreate_first_operator

CRITICALtool ↗

setup_status tool has empty input schema ({}). Tool accepts no parameters but description does not explain what it returns.

setup_status

HIGHresponse-shaper ↗

No output schemas documented for either tool. LLMs cannot plan downstream calls or extract required fields without knowing response structure.

setup_statuscreate_first_operator

HIGHtool-description ↗

Tool descriptions are HTTP route summaries, not LLM-optimized. 'GET /api/setup/status - Check if setup is needed...' is 60 chars but reads as API documentation, not agent guidance. Should explain WHEN to call, WHAT it returns, and WHY the agent would use it.

setup_statuscreate_first_operator

Recommendations

Implement MCP protocol: register tools via MCP's tool registration mechanism, not as HTTP routes. Use the MCP SDK for Python to expose tools with proper schemas and descriptions.
Document output schemas for both tools. Specify what fields setup_status returns (e.g., {setup_needed: boolean, bootstrap_token_active: boolean}) and what create_first_operator returns (e.g., {operator_id: string, session_token: string, email: string}).
Rewrite tool descriptions as LLM-optimized guidance. Example: 'setup_status: Check whether the system requires initial setup and if the bootstrap token is still valid. Call this first to determine if create_first_operator is available. Returns a boolean indicating setup state and token validity.' (120+ chars, action-oriented).
Expand parameter descriptions with format and validation rules. E.g., 'password: Operator password (minimum 8 characters, must include uppercase, lowercase, and digit). Used for authentication on subsequent logins.'
Add error handling guidance. Document what errors create_first_operator can return (invalid token, email already exists, weak password) and what the agent should do in each case (retry, ask user, suggest alternative).

Score history

Overall score trend

First recorded score · v2 rubric

39/100

Scored	Grade	Overall	Spec posture	Rubric
2026-09-23	F	39	2026-07-28+	v2