VoidPadding: Let [VOID] Handle Padding in Masked Diffusion Language Models so that [EOS] Can Focus on Semantic Termination
About
MDLMs generate text by denoising a preallocated masked response canvas, making response-length modeling central to instruction tuning. Existing MDLMs often inherit the autoregressive convention of using repeated \texttt{[EOS]} tokens for padding during instruction tuning, giving \texttt{[EOS]} a dual role as both a semantic terminator and a padding token. We show that this dual role is a root cause of \texttt{[EOS]} overflow under large-block decoding. To decouple these roles, we propose VoidPadding, which introduces \texttt{[VOID]} for padding and reserves \texttt{[EOS]} for termination. During inference, the learned \texttt{[EOS]} signal enables early stopping, while the learned \texttt{[VOID]} signal guides adaptive response canvas expansion. On Dream-7B-Instruct, VoidPadding improves the block-size-averaged four-task mean across mathematical reasoning and code generation benchmarks by \(+17.84\) points over the original model and \(+6.95\) points over RainbowPadding, while reducing decoding NFE by 55.7\% on average. Code is available at https://github.com/Haru-LCY/VoidPadding.
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Mathematical Reasoning | MATH 500 | Accuracy45.2 | 589 | |
| Code Generation | HumanEval | Accuracy60.98 | 212 | |
| Mathematical Reasoning | MATH 500 | Accuracy41.6 | 183 | |
| Mathematical Reasoning | GSM8K | Accuracy78.62 | 105 | |
| Code Generation | HumanEval | Score60.37 | 78 | |
| Code Generation | HEval | -- | 21 | |
| Code Generation | MBPP | Accuracy45.38 | 5 | |
| General Reasoning and Generation | GSM8K, MATH500, HumanEval, MBPP | Mean Accuracy51.34 | 5 | |
| Mathematical Reasoning | MATH500 | Accuracy39.6 | 5 | |
| Mathematical Reasoning | GSM8K | Accuracy (GSM8K)78.32 | 5 |