Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Introspective Diffusion Language Models

About

Diffusion language models promise parallel generation, yet still lag behind autoregressive (AR) models in quality. We stem this gap to a failure of introspective consistency: AR models agree with their own generations, while DLMs often do not. We define the introspective acceptance rate, which measures whether a model accepts its previously generated tokens. This reveals why AR training has a structural advantage: causal masking and logit shifting implicitly enforce introspective consistency. Motivated by this observation, we introduce Introspective Diffusion Language Model (I-DLM), a paradigm that retains diffusion-style parallel decoding while inheriting the introspective consistency of AR training. I-DLM uses a novel introspective strided decoding (ISD) algorithm, which enables the model to verify previously generated tokens while advancing new ones in the same forward pass. From a systems standpoint, we build I-DLM inference engine on AR-inherited optimizations and further customize it with a stationary-batch scheduler. To the best of our knowledge, I-DLM is the first DLM to match the quality of its same-scale AR counterpart while outperforming prior DLMs in both model quality and practical serving efficiency across 15 benchmarks. It reaches 69.6 on AIME-24 and 45.7 on LiveCodeBench-v6, exceeding LLaDA-2.1-mini (16B) by more than 26 and 15 points, respectively. Beyond quality, I-DLM is designed for the growing demand of large-concurrency serving, delivering about 3x higher throughput than prior state-of-the-art DLMs.

Yifan Yu, Yuqing Jian, Junxiong Wang, Zhongzhu Zhou, Donglin Zhuang, Xinyu Fang, Sri Yanamandra, Xiaoxia Wu, Qingyang Wu, Shuaiwen Leon Song, Tri Dao, Ben Athiwaratkun, James Zou, Fan Lai, Chenfeng Xu• 2026

Related benchmarks

TaskDatasetResultRank
Instruction FollowingIFEval
IFEval Accuracy84.7
625
General KnowledgeMMLU
MMLU General Knowledge Accuracy82.4
234
Mathematical ReasoningMATH 500
Accuracy (Acc)97.6
149
Mathematical ReasoningAIME 2024
Accuracy83.3
104
ReasoningARC Challenge
Accuracy96.8
93
Knowledge ReasoningMMLU
MMLU Knowledge Reasoning Accuracy86.8
65
Code GenerationLiveCodeBench v6
Accuracy57.1
58
Mathematical ReasoningGSM8K
Accuracy (Acc)95
42
Knowledge ReasoningMMLU-Pro--
40
Knowledge ReasoningGPQA
Accuracy58.7
18
Showing 10 of 12 rows

Other info

GitHub

Follow for update