Loading ToolBench
MCP server that integrates Slack and weather APIs with ChatGPT for summarizing Slack activity and providing weather information
The server defines 5 tools with visible implementations in Python. All tools have descriptions and input schemas with type definitions. However, several critical issues limit quality: (1) Tool naming is inconsistent, slack_list_channels uses 'list' but the function is 'list_slack_channels'; parameter naming is also inconsistent (channel_id vs state). (2) Descriptions are present but minimal (avg 62 chars), most lack WHEN to use guidance and prerequisites. (3) Schemas are well-typed but lack constraints (no min/max for limits, no enums for state codes). (4) Output schemas are completely undocumented, tools return strings without structure guidance. (5) Error handling returns unstructured error messages without recovery guidance. (6) No pagination despite limit parameters suggesting large result sets. The server appears functional but falls short of production-grade quality standards.
Get weather alerts for a US state.
Get weather forecast for a location.
Get recent messages from a Slack channel.
List all channels in the Slack workspace.
Send a message to a Slack channel.
No output schemas documented. All tools return unstructured strings instead of typed JSON objects. This forces LLMs to parse free-form text, wasting tokens and risking extraction errors. Example: slack_get_messages returns concatenated message strings with \n--- separators instead of a structured array of message objects with fields (user, timestamp, text, reactions, thread_info).
Parameter descriptions lack constraints and format guidance. Example: 'limit' parameters (slack_list_channels, slack_get_messages) have no min/max bounds in schema or description, source code enforces 1000 max, but this is invisible to LLMs. 'state' parameter in get_alerts accepts free-form string without enum constraint, LLM will hallucinate invalid state codes like 'California' instead of 'CA'.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 44 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 46 | - | v1 |
Descriptions are below recommended minimum (10-200 chars for clarity). Average across all tools is 53 chars. Descriptions lack WHEN to use guidance, prerequisites, and error recovery instructions. Example: slack_send_message (55 chars) does not clarify if it supports threads, formatted text, or retry behavior, forcing LLM to guess.
No pagination for tools with 'limit' parameters. slack_list_channels and slack_get_messages accept limits up to 1000 but return results as single flat strings, no next_cursor, offset, or total_count. If a workspace has thousands of channels or hundreds of messages, the tool can only return a truncated list or bloat context with thousands of results.
Error messages are unstructured and provide no recovery guidance. Example: slack_get_messages returns 'Failed to get channel messages: {error}' (raw API error), does not tell LLM if error is retryable, if it should try a different channel, or what to do next. Pattern: recovery-guide requires 'User not found. Try search_users() with a partial name.' style guidance.
Tool naming is inconsistent with action verbs. slack_list_channels uses 'list' (good) but get_alerts and get_forecast use generic 'get' (poor). According to baselines, specific verbs like 'list_', 'search_', 'get_' are expected, but 'get' alone is ambiguous, 'get_weather_alerts' would be clearer. Parameter naming also inconsistent: slack tools use 'channel_id', weather tools use 'state' and bare coordinates (no '_id' suffix).
Missing natural-language identifier support. slack_send_message requires 'channel_id' (opaque ID like 'C12345') but users say 'send a message to #general'. No lookup tool (e.g., slack_find_channel_by_name) provided, agent must already know channel IDs. Pattern: mxe:natural-identifiers recommends accepting usernames/display names and resolving to IDs internally.
Tool outputs may leak secrets or internal details. slack_get_messages calls users.info API for each user and returns real_name and name fields without filtering. If user names contain PII or if the returned fields include profile URLs or user IDs, these enter the LLM context and risk being echoed back to the user or logged. Pattern: secret-injection requires tool responses to strip internal secrets.