Static source inference · medium confidence · detected: Logging
Deprecated protocol patterns detected
Summary
The server provides 8 tools with reasonable structure and naming conventions. Most tools start with action verbs (get_, create_, update_, delete_, run_) which is positive. However, there are significant gaps: (1) descriptions are present but many are terse (under 100 chars for several tools); (2) parameter descriptions exist but lack detail about constraints, formats, and valid ranges; (3) no documented output schemas visible in the provided code; (4) error handling appears minimal based on code inspection; (5) the tools themselves are properly namespaced but lack composition patterns that would support multi-step workflows. The tools are well-named and generally follow single-responsibility principles, but lack the LLM-optimized clarity and constraint documentation needed for A-grade production use.
No documented output schemas. Tools return data but response structure is not defined in visible code. LLMs cannot predict what fields are available for downstream processing or chaining.
Parameter descriptions lack constraint details. E.g., 'modifications' in clone_scenario has no description of what fields it accepts. 'build_mode' enum values are documented but format/pattern constraints for strings like 'project_key', 'scenario_id', 'recipe_name' are absent.
Destructive and write operations lack confirmation or dry-run patterns. delete_recipe is marked DESTRUCTIVE but no error handling guidance or confirmation step is visible in code. Agents could accidentally delete recipes without recovery options.
Recommendations
Document all output schemas explicitly. For each tool, specify the response structure: field names, types, and whether fields are optional. Example for get_recipe_info: 'Returns {recipe_id: string, recipe_name: string, recipe_type: string, inputs: array<string>, outputs: array<string>, code: string|null, created_at: ISO8601String, modified_at: ISO8601String}'.
Expand parameter descriptions to include constraints, formats, and valid ranges. For 'project_key', add: '2-10 uppercase alphanumeric characters (e.g., PROJ_001). Must exist in the Dataiku instance.' For 'modifications' in clone_scenario, specify: 'Dict with optional keys: {name: string, description: string, tags: array<string>, triggers: object}. Unspecified keys retain source values.'
Add error recovery guidance to all tool descriptions. Example: 'If the scenario_id is not found, try calling list_scenarios() to discover available scenario IDs.' Or: 'Errors: scenario_not_found (suggest discovery), permission_denied (request elevated credentials), api_error (retryable with exponential backoff).'
Implement and document a confirmation pattern for destructive operations. For delete_recipe, add an optional 'confirm: bool' parameter or a 'dry_run: bool' parameter that returns what would be deleted without executing. Document: 'To prevent accidental deletion, this tool requires confirm=true in production; dry_run=true shows impact without deletion.'
Distinguish overlapping tools in descriptions. For get_scenario_logs vs. get_scenario_steps: 'get_scenario_logs: Retrieve execution logs, error messages, and runtime diagnostics from a specific scenario run. Use this to debug failed runs. get_scenario_steps: Retrieve the step configuration, code, and trigger settings of a scenario definition. Use this to understand the scenario structure or replicate it.'
Score history
Overall score trend
↓ 33 points across a rubric change (v1 → v2)
8/100
Scored
Grade
Overall
Spec posture
Rubric
2026-09-22
F
8
<=2025-11-25
v2
2026-03-09
F
41
-
v1
delete_recipedestructiveauthsource verified
Delete a recipe from a project.
delete_scenariodestructiveauthsource verified
Delete a scenario from a project.
get_code_environmentsread onlyauthsource verified
List available Python/R environments.
get_connectionsread onlyauthsource verified
List available data connections.
get_project_variablesread onlyauthsource verified
Get project-level variables and configuration.
get_recipe_coderead onlyauthsource verified
Extract actual Python/SQL code from recipes.
get_scenario_logsread onlyauthsource verified
Get detailed run logs and error messages for failed scenarios.
get_scenario_stepsread onlyauthsource verified
Get detailed step configuration including Python code.
Error messages in code are minimal. Examples show generic catches (e.g., 'except Exception as e') with no categorization of retryable vs. fatal errors. No guidance on what the agent should do next (e.g., 'Try search_scenarios() first').
Tool descriptions do not distinguish when to call this vs. similar tools. E.g., get_scenario_logs vs. get_scenario_steps, both read from scenarios but have subtly different purposes that are not clearly stated.
Composition patterns not evident. Tools do not appear to return IDs that naturally chain into downstream calls. E.g., create_recipe description does not state whether it returns a recipe_id usable in update_recipe or run_recipe.
Short descriptions on several tools. 'Get detailed step configuration including Python code.' (get_scenario_steps) is only 55 chars, below baseline (194 chars average for A+ tools). Descriptions do not explain WHEN or WHY to call these tools.
get_scenario_stepsget_scenario_logsdelete_recipe
Return IDs in create/update responses that enable chaining. Ensure create_recipe returns {recipe_id, recipe_name, ...} so the agent can immediately call run_recipe(recipe_name=<returned_value>). Document in the response schema.
Add idempotency and side-effect clarity to descriptions. For write tools: 'Note: This operation is not idempotent. Repeated calls create duplicate scenarios/recipes. Agents should store the returned ID and reuse it rather than retrying creates.' Or offer an 'upsert_scenario' variant.
Implement per-item error details for any tools returning multiple items (e.g., list_scenarios if it exists). Instead of 'Error: 3 scenarios failed', return [{scenario_id, status, error: 'access_denied'}, ...] so the agent can retry specific items.
Add pagination support to list operations if they exist. If getting scenario logs returns multiple runs, document: 'Returns {total_runs: int, runs: [{run_id, start_time, outcome, ...}], next_cursor: string|null}. Use next_cursor in a follow-up call to fetch remaining runs.' Cap results at 20-50 items per call.