Loading ToolBench
Three FastMCP servers for NYC subway trip planning, service status, and geocoding: alerts (live service status), routing (Dijkstra trip planning), and geocode (place/station resolution)
Strong foundation with 9 well-named tools, comprehensive descriptions (avg 180 chars), and explicit JSON schemas. All tools are READ_ONLY with proper annotations. Key gaps: output schemas are not formally documented in the code (inferred from docstrings); error handling lacks recovery guidance; some parameter descriptions could be more prescriptive about formats/constraints. Tool names follow verb_noun pattern consistently (get_*, list_*, plan_*, resolve_*, geocode_*, nearest_*). Parameters accept flexible types (string|integer) to match LLM behavior, which is pragmatic but underdocumented.
Resolve a free-text place or landmark to coordinates and the nearest station. Tries a landmark table first (e.g. "Empire State Building"), then station names. Returns {lat, lng, source, matched, nearest_station{...}}.
Service status for a single subway line (e.g. "A", "7", "Q"). Accepts a number as well as a string: models routinely send the 7 train as the integer 7, and a schema that rejects it just burns retries.
Current service status for every NYC subway line. Returns a summary string plus a per-line map of {severity 0-3, status, message, updatedAt}. Severity: 0 good, 1 delays, 2 service change/planned work, 3 suspended.
Current elevator/escalator outages, optionally filtered by station name. Matching is normalized ("Times Square" finds "Times Sq-42 St"). Returns {"available": False} when the live outage feed can't be reached, so the caller never reports "no outages" it didn't actually observe.
List modeled stations, optionally only those served by a given line. The network has 475 stations, so results are capped (default 50) — an unfiltered dump is ~28 KB of context for no benefit. `count` is the true number of matches; narrow with `on_line` rather than raising the limit.
Output schemas not formally documented. Docstrings describe return structure (e.g., 'per-line map of {severity, status, message, updatedAt}') but no JSON Schema definitions visible in code. LLMs cannot reliably parse unstructured output descriptions.
Parameter type coercion (string|integer) is pragmatic but underdocumented. Docstrings mention 'models routinely send the 7 train as integer 7' but do not explain why both types are accepted or how the tool normalizes them. This forces LLMs to guess at the rationale.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 65 | 2026-07-28+ | v2 |
Nearest modeled subway station to a latitude/longitude, with distance_km. Coordinates arriving as strings are coerced — same reason as `line`.
Plan the fastest subway trip. Use this for any trip question. `origin` and `destination` are free text — a station name, a landmark, a neighborhood, or a station id returned by another tool. Resolves each endpoint to a station (snapping a landmark such as "Williamsburg" to its nearest station, noted in the summary), then runs Dijkstra. If either endpoint is ambiguous, returns {"needs_disambiguation": True, ...} with candidates so the caller can clarify. Otherwise returns a full itinerary: numbered legs, total minutes, stop count, transfer count, and a summary sentence.
Plan a trip between two known station ids. Prefer `plan_trip`, which takes ids as well as names; this exists for callers that already have ids.
Resolve free text to the best-matching subway station. Returns the matched station, a candidate list, and an `ambiguous` flag when several stations share the name (e.g. "125 St").
Error handling lacks recovery guidance. Docstrings mention degradation ('returns {available: False} when feed unavailable') but do not tell the LLM what to do next. Should include: 'If available=False, retry in 30s or ask user for cached data.'
plan_trip returns disambiguation structure ({needs_disambiguation: True, candidates: [...]}) but the docstring does not explain the exact schema of candidates or how the LLM should handle the ambiguous case. Incomplete specification invites misuse.
list_stations caps results at 50 (default) with a 200-item hard limit, but the description does not explain pagination strategy or how to fetch the next page. No offset/cursor parameter visible. Users cannot iterate over all 475 stations.