Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

VoidPadding: Let [VOID] Handle Padding in Masked Diffusion Language Models so that [EOS] Can Focus on Semantic Termination

About

MDLMs generate text by denoising a preallocated masked response canvas, making response-length modeling central to instruction tuning. Existing MDLMs often inherit the autoregressive convention of using repeated \texttt{[EOS]} tokens for padding during instruction tuning, giving \texttt{[EOS]} a dual role as both a semantic terminator and a padding token. We show that this dual role is a root cause of \texttt{[EOS]} overflow under large-block decoding. To decouple these roles, we propose VoidPadding, which introduces \texttt{[VOID]} for padding and reserves \texttt{[EOS]} for termination. During inference, the learned \texttt{[EOS]} signal enables early stopping, while the learned \texttt{[VOID]} signal guides adaptive response canvas expansion. On Dream-7B-Instruct, VoidPadding improves the block-size-averaged four-task mean across mathematical reasoning and code generation benchmarks by \(+17.84\) points over the original model and \(+6.95\) points over RainbowPadding, while reducing decoding NFE by 55.7\% on average. Code is available at https://github.com/Haru-LCY/VoidPadding.

Chunyu Liu, Zhengyang Fan, Kaisen Yang, Alex Lamb• 2026

Related benchmarks

TaskDatasetResultRank
Mathematical ReasoningMATH 500
Accuracy45.2
589
Code GenerationHumanEval
Accuracy60.98
212
Mathematical ReasoningMATH 500
Accuracy41.6
183
Mathematical ReasoningGSM8K
Accuracy78.62
105
Code GenerationHumanEval
Score60.37
78
Code GenerationHEval--
21
Code GenerationMBPP
Accuracy45.38
5
General Reasoning and GenerationGSM8K, MATH500, HumanEval, MBPP
Mean Accuracy51.34
5
Mathematical ReasoningMATH500
Accuracy39.6
5
Mathematical ReasoningGSM8K
Accuracy (GSM8K)78.32
5
Showing 10 of 10 rows

Other info

Follow for update