Loading ToolBench
An MCP server around SU2 workflows (mesh presets + open-ended density refinement + Cauchy-based inner convergence reporting)
su2-mcp demonstrates solid fundamental quality with 17 well-defined tools covering a coherent domain (SU2 CFD workflow). All tools have descriptions and input schemas visible in source. Naming is consistently verb-noun (get_, create_, update_, list_, parse_, run_, etc.). However, there are systematic gaps in parameter descriptions, output schema documentation, and error recovery guidance. The server implements fastmcp on STDIO, limiting remote accessibility. No tool annotations (readOnlyHint/destructiveHint/idempotentHint) despite risk labels being provided in the rubric. Descriptions range from adequate (60-80 chars for shorter ops) to comprehensive (200+ chars for complex mesh/solver ops), but parameter-level descriptions are often minimal. Error handling returns structured error dicts but lacks actionable recovery guidance (e.g., 'try search_users()' patterns). Output schemas are inferred from code returns but not formally documented in tool registration.
Analyze the mesh attached to a session and return diagnostics. Reports element counts by type, node count, boundary marker summary, and estimated solver runtime scaling factors to help diagnose latency.
Run SU2_DEF to create a deformed mesh.
Generate a 3D SU2 mesh from a STEP file and attach it to the given session. Uses a .geo template that merges the STEP, builds an aircraft volume (surface loop -> volume), creates a farfield box, and meshes the fluid domain with FARFIELD and WALL markers. Requires the `gmsh` CLI to be on PATH.
Return raw config text for the session.
Return a base64-encoded slice of a result file.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) in tool registration despite clear risk labels (READ_ONLY, WRITE, DESTRUCTIVE, WRITE). Tools like close_su2_session and run_su2_solver are destructive but not marked for LLM safety awareness.
Error responses lack actionable recovery guidance. Functions like parse_config and update_config_entries return generic error dicts (_error helper) without suggesting next steps (e.g., 'SessionNotFound: call list_sessions() or create_su2_session()' would guide the LLM).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 64 | 2026-07-28+ | v2 |
| 2026-03-09 | C | 60 | - | v1 |
List result files produced within a session directory.
Parse the session configuration into key/value entries.
Read a subset of a history CSV file.
Run a SU2 solver process and capture output metadata.
Sample surface solution fields from a CSV-like file.
Persist a mesh file for the session and update the config.
Update configuration entries for a session.
Output schemas not formally documented in tool registration. While code returns structured dicts (e.g., {'config_text': ...}, {'entries': ...}), the MCP tool schema does not include outputSchema declarations. LLMs cannot plan downstream tool chains without knowing what fields are returned.
No documented dependencies or parameter relationships. For example, generate_mesh_from_step requires a valid session_id (created via create_su2_session), and gmsh must be on PATH. These constraints should be stated in parameter and tool descriptions.
Numeric parameters lack explicit ranges. For example, gmsh_timeout_seconds, capture_log_lines, max_runtime_seconds, max_bytes, max_rows, and max_points have no documented min/max. This invites LLMs to pass absurd values (e.g., max_bytes=999999999) that break execution.
No tool for discovering available sessions. Agents can create and close sessions but have no list_sessions() or get_active_sessions() tool. This forces LLMs to guess session_id values or fail silently when a session doesn't exist.