Loading ToolBench
Company-agnostic SRE operations platform with AI-powered monitoring, 33+ MCP tools, plugin architecture for multi-company support, vector store intelligence, and comprehensive infrastructure observability
37 tools with significant structural gaps. Most tools (25+) lack complete input schemas or have minimal descriptions. Many tool names are generic (check_*, get_*, search_*) without clear action verbs. Descriptions range from adequate (analyze_alerts, triage_service) to trivial (find_operations_scripts, get_affected_servers). No visible output schemas documented. Error handling and recovery guidance absent. Tool composition is poor, multiple tools do similar things (check_jboss_health, check_database_health, check_haproxy_health, check_messaging_health are near-duplicates). Parameter descriptions often missing or generic. No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite 8 WRITE-risk tools. Security concerns: execute_remote_command and execute_on_inventory are high-risk with minimal parameter validation guidance.
Trigger aggregation of data from Silver to Gold layer. Calculates business-level metrics and KPIs.
Analyze recent alerts from the Integral Forex platform. Provides AI-powered insights on alert patterns, hotspots, and recommendations. Use this when users ask about alerts, incidents, or system health. Examples: - "What alerts happened today?" - "Show me Singapore alerts" - "Analyze recent error patterns"
Check health status of database servers.
Check health status of HAProxy load balancers.
Check health status of JBoss application servers.
Check health status of messaging systems.
12 tools with trivial or missing descriptions (<30 chars): find_operations_scripts, get_affected_servers, check_jboss_health, check_database_health, check_haproxy_health, check_messaging_health, ingest_opsgenie_data, ingest_monitoring_urls, sync_operations_scripts, get_ops_insights, run_hourly_ingestion, ingest_data_server. LLMs cannot determine when to select these tools.
Four near-duplicate health check tools (check_jboss_health, check_database_health, check_haproxy_health, check_messaging_health) with identical schemas and minimal descriptions. Violates single-responsibility principle. LLM cannot distinguish when to use which.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 46 | <=2025-11-25 | v2 |
Check health status of a server using Prometheus metrics or SSH commands.
Execute operations on servers from the inventory.
Execute a command on a remote server via SSH.
Search for operations scripts and automation tools in the repository.
Get list of servers affected by an alert or incident.
Get detailed incident history with timeline of events. Shows which tools were used, what docs were consulted, resolution time. Essential for post-mortems and learning from past incidents. Examples: - "Show me incident INC-2025-009 timeline" - "What was the resolution for incident INC-2025-008?" - "Incident history for last incident"
Get live metrics for services.
Get monitoring dashboard URLs for a service or server.
Get operational insights and recommendations.
Retrieve runbook documentation from Confluence for a service. Provides troubleshooting steps, common issues, and escalation procedures. Use this when users need operational documentation or troubleshooting guidance. Examples: - "Show me the runbook for sgfxiadp14" - "How do I troubleshoot ppfxiadp75?" - "What's the escalation procedure for LN services?"
Ingest data from Operations Data Server (data1.integral.com).
Ingest raw data into the Bronze layer of the Datalake. Use this to save raw logs, metrics, or event data.
Ingest monitoring URLs into the knowledge base.
Ingest alert data from OpsGenie into the knowledge base.
Manually fetch and ingest alerts from Slack #fxcloudwatch-alert channel. Updates the local alert database with recent Slack messages. Use this to refresh alert data from Slack before analysis. Examples: - "Refresh alerts from Slack" - "Fetch latest alerts" - "Update alert data from Slack"
Parse hostname to get infrastructure information (environment, location, system type). Integral infrastructure uses hostname conventions to encode environment and location. Use this to understand which environment/datacenter a server belongs to and determine alert severity. Examples: - "What environment is ppfxiadp1 in?" - "Parse hostname dmfxidb1" - "Get server info for tpfxiapp2" Hostname format: [location][system][number] - pp/lp/tp/sp = Production (London/Tokyo/Singapore) - dm = Demo - fxi = FX Integral, adp = Adapter, db = Database, clob = CLOB
Trigger processing of data from Bronze to Silver layer. Cleans, normalizes, and enriches raw data.
Query alert pattern intelligence from multi-week historical analysis. Get information about error codes, high-chatter services, regional patterns, and triage recommendations. Based on comprehensive alert pattern analysis across all regions and services. Examples: - "What is CLB-XX-029?" - "Is SUADAP4 a known noisy service?" - "Get triage recommendation for FCC-XX-008 on UA-GRP2-S3 with 60 occurrences" - "Show me patterns for PROD-LDN region"
Query metrics from the Datalake (Bronze, Silver, or Gold). Layers: - Bronze: Raw ingested data - Silver: Cleaned and normalized data - Gold: Aggregated business metrics
Execute a PromQL query against a regional Prometheus instance.
Run hourly data ingestion tasks.
Semantic search across company documentation (Confluence runbooks, procedures, best practices). Finds relevant docs based on MEANING, not just keywords. Always check this before making recommendations to follow company patterns. Examples: - "How do we handle database connection issues?" - "What's the process for deploying to production?" - "Show me escalation procedures"
Search monitoring URLs from the knowledge base.
Search operations messages and communications.
Search OpsGenie alerts and incidents.
Search for operations scripts.
Interactive shift check procedure for SRE on-call. Guides through standardized checks at shift start/end. Based on company runbooks and best practices. Examples: - "Start shift check" - "Run shift handoff procedure" - "What should I check at beginning of shift?"
Sync operations scripts from repository.
Tail a log file on a remote server.
Test SSH connectivity to a remote server.
Perform incident triage for a specific service. Correlates recent incidents with alerts and provides actionable recommendations. Use this when users ask about a specific service's health or incidents. Examples: - "Triage sgfxiadp14" - "What's wrong with ppfxiadp75?" - "Analyze incidents for the NY trading service"
Two high-risk WRITE tools (execute_remote_command, execute_on_inventory) lack parameter validation guidance, error recovery hints, and confirmation/dry-run patterns. No tool annotations (destructiveHint). Agents could execute arbitrary commands without safeguards.
No output schemas documented for any tool. LLMs cannot plan downstream tool calls or extract required fields (e.g., incident_id, service_name) for chaining. Violates response-shaper pattern.
8 tools with empty or minimal input schemas (ingest_opsgenie_data, ingest_monitoring_urls, sync_operations_scripts, get_ops_insights, run_hourly_ingestion, ingest_data_server, process_datalake_silver, aggregate_datalake_gold). No parameter descriptions. Schema score must be 0-30.