Garak (LLM vulnerability scanner)

Input Type:#

This parser imports the JSON Lines hit log produced by garak, NVIDIA’s LLM vulnerability scanner.

A garak run writes garak.<run_id>.hitlog.jsonl alongside its report.jsonl. Every line in the hit log is, by construction, a detector hit, so each record is mapped to a DefectDojo Finding. Upload the *.hitlog.jsonl file (not report.jsonl).

Tested against the garak 0.15.x hit-log schema (garak/evaluators/base.py).

Things to note about the Garak parser:#

• Aggregation: hits for the same probe, target (generator), and detector are aggregated into a single Finding, with nb_occurences reflecting the number of hits and the most severe rung retained.
• Severity is derived from the detector score (0.0-1.0) and adjusted by probe family. Active-attack / code-execution / jailbreak families (e.g. promptinject, dan, malwaregen, xss) are nudged up one rung; content/quality families (e.g. continuation, misleading, toxicity) are nudged down one rung. Note that many garak detectors are string/word-list matchers that emit a binary score of 1.0, so most real hits land in the upper severity bands.
• CWE is mapped from the probe family as a starter mapping (refined over time):
  • prompt-injection families (promptinject, dan, latentinjection, goodside) -> CWE-1427 (Improper Neutralization of Input Used for LLM Prompting)
  • xss -> CWE-79
  • leakreplay, divergence -> CWE-200
  • all other families -> CWE-1426 (Improper Validation of Generative AI Output)
• A hit log with no detector hits yields no findings. Lines that are not hit records (anything without a probe field, such as run/config metadata) are ignored.

JSON Lines Format:#

The parser accepts a .jsonl hit log. Each line is one hit record with fields including goal, prompt, output, triggers, score, probe, detector, and generator. The prompt and output values are serialized garak conversation/message objects (nested dicts), from which the parser extracts the displayed text.

Sample Scan Data#

Sample scan data for testing purposes can be found here.

Deduplication#

The “Garak Scan” scan type uses the hash_code deduplication algorithm with the following fields:

• title (the garak probe and its goal)
• component_name (the scanned model / generator)

description and severity are intentionally excluded from the hashcode. description holds the specific prompt and model output for the hit, which garak samples non-deterministically on each run. severity is an aggregate value — the most severe rung seen across a probe’s occurrences — so it shifts as the occurrence set changes between scans. Including either would stop the same weakness from deduplicating across repeated scans of the same model.