GenAI-Accelerated Career Advancement platform with analytics and career agent tools
NextRole exposes 7 tools across two agents (analytics and career). Tool definitions are visible in backend/agents/*/tools.py with descriptions and input schemas. However, critical gaps undermine quality: (1) Descriptions lack LLM-optimized structure, they are verbose (200-400+ chars) and bury key intent under implementation details. E.g., describe_data's description is 380+ chars explaining caching behavior before stating the core action. (2) Parameter descriptions are present but often generic or incomplete, 'table' in describe_data lacks guidance on when to omit vs. provide; 'search_depth' in web_search has no explanation of what 'advanced' vs 'basic' actually retrieves. (3) Output schemas are entirely undocumented, no tool declares what fields it returns, forcing LLMs to infer structure. (4) Error handling is absent, no guidance on retryability, user-fixable errors, or recovery paths. (5) Naming is acceptable (verb_noun pattern) but some tools are vague: 'web_extract' duplicates 'web_search' intent without clear distinction. Per-tool average: 52/100.
Build a chart from SQL or a pre-built Plotly figure and publish it to the dashboard.
Describe the warehouse: what tables exist and what their columns mean. Call this before writing your first query of a conversation, and again for any table you have not yet inspected. The warehouse's column semantics are not guessable from names alone — `owner` has a special `'default'` value, `interrupted` runs are not failures, and estimated cost is a list-price floor, all of which change what a number means. With no argument: the layer map, every mart with its row count and what one row represents, and the metric views already defined. With a table name (`"fct_run"` or `"nextrole_marts.fct_run"`): every column with its type, meaning and PII class, the sort key to filter on, and the metrics Cube defines over that table.
List files in a directory with size and modification time, newest first. Use this when you need recency or size info, or want results ordered by modification time. For a quick path-only listing, prefer the built-in `ls`. Works on any path the agent can reach — `/upload/`, `/processed/`, `/research/`, `/interview_prep/`, `/workspace/`, etc.
Parse a document with LlamaParse and persist the result as markdown. Works on any document the agent's filesystem can read: PDFs, DOCX, PPTX, images, etc. Common source dir is `/upload/` (user uploads), but any readable backend path is supported. The parsed markdown is written directly to `output_path` — you do NOT need to call `write_file` afterwards.
No output schemas documented. Tools return data but LLMs cannot infer field names, types, or structure. E.g., run_sql returns results but no schema specifies columns, types, or pagination fields.
Descriptions are verbose (200-400+ chars) and bury intent under implementation details. E.g., describe_data spends 150+ chars on caching behavior before stating the core action. LLM-optimized descriptions should be 50-200 chars, action-first.
Parameter descriptions are generic or incomplete. 'search_depth' in web_search has no explanation of what 'advanced' vs 'basic' retrieves. 'table' in describe_data lacks guidance on when to omit. LLMs cannot infer parameter semantics from names alone.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 51 | 2026-07-28+ | v2 |
Run one read-only SQL query against the ClickHouse warehouse. ClickHouse dialect. Only the marts and staging databases are readable, and only for reading — the connection has no write privileges at all. Always qualify table names (`nextrole_marts.fct_run`) and filter on the table's sort key, which `describe_data` reports. Results are capped for display. When a query has no LIMIT, one is added and the result says so, so a partial answer is never mistaken for a complete one. Use `save_as` only when a chart or analysis needs more rows than are worth reading: it writes the full result to a file for a Python script run through `execute`, and returns the path. Do not use it to "look at" data — read the table this tool prints instead.
Search the web for current information.
Search the web for current information.
No error handling guidance. Tools provide no recovery hints, retryability classification, or actionable error messages. E.g., run_sql may fail on invalid SQL but offers no guidance on what to fix.
web_search and web_extract have overlapping intent without clear distinction. LLMs may conflate them or choose incorrectly. Naming should disambiguate: e.g., 'search_web' vs 'extract_web_content'.