Garak integration through the Model Context Protocol - enables vulnerability scanning of LLM models using the Garak security testing framework
The Garak MCP server has significant definition quality gaps. While 5 tools are explicitly registered with descriptions, most descriptions are generic and lack the specificity needed for LLM tool selection. Parameter descriptions are minimal or missing. Output schemas are not documented. The tool set is focused but lacks idempotency guidance and error recovery patterns. Average tool score: 42/100.
Get the report of the last run. Returns: str: The path to the report file.
List all available Garak attacks. Returns: list: A list of available probes / attacks.
List all available model types. Returns: list[str]: A list of available model types.
List all available models for a given model type. Those models can be used for the attack and target models. Args: model_type (str): The type of model to list (ollama, openai, huggingface, ggml) Returns: list[str]: A list of available models.
Run an attack with the given model and probe which is a Garak attack. Args: model_type (str): The type of model to use. model_name (str): The name of the model to use. probe_name (str): The name of the attack / probe to use. Returns: list: A list of vulnerabilities.
Output schemas undocumented. Tools return untyped responses (e.g., run_attack returns 'list: A list of vulnerabilities' with no field structure). LLMs cannot plan downstream calls or extract specific data from responses.
Parameter descriptions are absent or minimal. 'model_type' in list_models has a description, but model_name and probe_name in run_attack lack clarity on expected format, validation rules, or how to obtain valid values.
No enum constraints for model_type. Parameter accepts 'ollama', 'openai', 'huggingface', 'ggml' but is declared as free-form string. LLMs will hallucinate invalid values.
No error handling guidance. run_attack may fail (invalid model, probe not found, Garak timeout) but tool description offers no recovery path. LLM receives raw output with no actionable next step.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 52 | 2026-07-28+ | v2 |
| 2026-03-09 | D | 51 | - | v1 |
Destructive operation (run_attack) lacks dry-run, confirmation, or idempotency guidance. Description does not warn that attacks may have side effects or repeat calls. Agents may not realize they are executing security tests multiple times.
Discovery tools (list_models, list_garak_probes) lack dependency hints. No guidance on when to call list_model_types before list_models, or how to use list_garak_probes output as input to run_attack.
get_report returns a file path string with no metadata. LLM cannot know if file exists, when it was created, or how to retrieve it. Description says 'path to report file' but does not explain format or access method.