Loading ToolBench
Strong tool coverage with clear action-verb naming (ctrace_symbols, ctrace_attach, ctrace_syscall_summary, etc.). All 20 tools have descriptions ranging 80-250 characters, well within the 10-1024 baseline. Input schemas are visible with proper type declarations (string, integer, float, array). However, output schemas are NOT documented, the code returns `dict` with no structure guidance, forcing LLMs to infer response fields. Parameters are reasonably constrained with sensible defaults (duration=5.0, top_n=20, threshold_us=1000). Session management is clean with optional session_id pattern. Risk classifications (READ_ONLY, WRITE, DESTRUCTIVE) hint at side effects but are not formalized as tool annotations in the MCP spec. Error handling is absent from visible code, no guidance on recovery, retryability, or actionable messages. Security appears solid (read-only tools dominate; no secrets exposed), but audit logging is not visible. Composition is strong, each tool has one clear job and chains naturally (define_tick → tick_summary/outliers/compare). Tool names follow verb_noun convention well.
Top allocation call sites by count and total bytes. Requires binary with symbols.
malloc/free/realloc rates, size distribution, net allocation rate (pressure indicator).
Attach to a running process by PID. Creates a tracing session. Validates the PID exists and returns process info.
Define a recurring loop by its entry function (e.g., physics_update, control_loop). Multiple independent loops can be defined. Used by tick analysis tools.
I/O operations exceeding latency threshold. Shows which files/sockets are slow.
Read/write ops by file descriptor, bytes transferred, latency. Identifies I/O-heavy FDs.
Output schemas are completely undocumented. All tools return `dict` with no field documentation, forcing LLMs to infer response structure and increasing hallucination risk.
No error handling guidance visible. Tools lack actionable error messages with recovery steps, retryability classification, or invalid-input explanations. LLMs will not know how to respond to failures.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 60 | 2026-07-28+ | v2 |
| 2026-03-09 | D | 57 | - | v1 |
Destroy a tracing session. Does NOT kill the target process.
Launch a binary with optional args and create a tracing session. Returns the session ID and PID of the launched process.
Mutex/futex waits exceeding threshold. Shows which locks cause contention.
Off-CPU flame-graph data: why threads are sleeping and for how long.
Per-thread: on-cpu time, off-cpu time, context switches, wakeup latency.
List all active tracing sessions with their PIDs and status.
Get process status: alive, CPU%, memory, thread count, open FDs.
List probeable function symbols from the session binary. Returns mangled name, demangled name (C++ aware), probe_pattern (ready to use in dtrace/bpftrace scripts), and tick_candidate flag. Searches both mangled and demangled names. For C++ symbols, probe_pattern is a glob like *ClassName*Method* that avoids dtrace quoting issues.
Top syscalls by count and cumulative latency. Shows histogram of latencies per syscall.
Trace specific syscalls (or all slow ones). Returns individual events with args, return value, latency.
List all threads of the traced process with names and CPU times. Thread names come from pthread_setname_np and can be passed directly to ctrace_define_tick as thread_filter to scope tick analysis to a specific thread.
Compare fast ticks (p50) vs slow ticks (p99+). Shows what differs: extra syscalls, allocations, lock waits.
Ticks exceeding deadline. For each outlier: what happened (syscalls, allocs, scheduling delays).
Per-tick stats: min/median/p99/max duration, syscall count, alloc count, off-cpu time.
Tool annotations (readOnlyHint, destructiveHint, idempotentHint) are not declared in MCP schema. Risk classifications exist in metadata ('Risk: READ_ONLY', 'Risk: DESTRUCTIVE') but are not formalized for MCP protocol compliance.
ctrace_sessions() has minimal description ('List all active tracing sessions...'), only 47 characters, borderline for LLM understanding. Could benefit from WHEN to call it and what structure to expect.
Parameter constraints are implicit. E.g., top_n defaults to 20 but no min/max specified. threshold_us defaults to 100 or 1000 depending on tool, but no range guidance. LLMs may pass absurd values (negative, zero, or gigantic numbers) without validation.