Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Diffusion LLMs are Natural Adversaries for any LLM

About

We introduce a novel framework that transforms the resource-intensive (adversarial) prompt optimization problem into an \emph{efficient, amortized inference task}. Our core insight is that pretrained, non-autoregressive generative LLMs, such as Diffusion LLMs, which model the joint distribution over prompt-response pairs, can serve as powerful surrogates for prompt search. This approach enables the direct conditional generation of prompts, effectively replacing costly, per-instance discrete optimization with a small number of parallelizable samples. We provide a probabilistic analysis demonstrating that under mild fidelity assumptions, only a few conditional samples are required to recover high-reward (harmful) prompts. Empirically, we find that the generated prompts are low-perplexity, diverse jailbreaks that exhibit strong transferability to a wide range of black-box target models, including robustly trained and proprietary LLMs. Beyond adversarial prompting, our framework opens new directions for red teaming, automated prompt optimization, and leveraging emerging Flow- and Diffusion-based LLMs.

David L\"udke, Tom Wollschl\"ager, Paul Ungermann, Stephan G\"unnemann, Leo Schwinn• 2025

Related benchmarks

TaskDatasetResultRank
Jailbreak AttackHarmBench (test)
ASRHB89.58
276
Jailbreak AttackSTRONGREJECT (held-out behaviors)
ASR (0.5 threshold)100
186
Jailbreak AttackStrongReject (test)
Score73.81
64
Jailbreaking Attack SuccessSTRONGREJECT 40 held-out behaviors
EVUS85
62
Jailbreak Attack SuccessSTRONGREJECT 60 behaviors (train)
EVUS82
62
Jailbreak Attack SuccessSTRONGREJECT (train)
ASR (0.5)100
62
Jailbreak RobustnessSTRONGREJECT (train)
EVUS82
62
Adversarial AttackJailbreakBench 50% stratified per-category sample (48 requests)
HB ASR10.42
32
Jailbreak AttackHarmful Prompts model-averaged
Model Averaged ASR (0.8)43.3
8
Showing 9 of 9 rows

Other info

Follow for update