Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Dasheng AudioGen: A Unified Model for Generating Coherent Audio Scenes from Text

About

Audio generation has long been fragmented, with speech, music, and sound effects produced by domain-specific models that fail to jointly generate coherent audio scenes from a single description. The key obstacles are insufficient fine-grained supervision for real-world mixed audio and limited acoustic representations for modeling concurrent audio components. We present Dasheng AudioGen, a unified framework for generating general mixed-audio scenes from text. Dasheng AudioGen introduces structured multi-view captions, which explicitly decouple complex acoustic scenes into complementary description views, thereby enabling fine-grained control over audio layers. Furthermore, we employ a high-dimensional unified semantic-acoustic representation as the shared latent space. It injects semantic priors that facilitate cross-modal training convergence, while its high-dimensional feature space provides sufficient capacity to disentangle and fuse concurrent audio components effectively. With these designs, a simple flow-matching DiT achieves high-quality end-to-end audio scene generation. We also establish a comprehensive evaluation pipeline for audio scene generation. Experiments demonstrate that Dasheng AudioGen achieves performance approaching real-world recordings in mixed-audio categories, while remaining competitive with specialized models in single-type generation tasks. Demos are available at https://nieeim.github.io/Dasheng-AudioGen-Web/.

Jiahao Mei, Heinrich Dinkel, Yadong Niu, Xingwei Sun, Gang Li, Yifan Liao, Jiahao Zhou, Junbo Zhang, Jian Luan, Mengyue Wu• 2026

Related benchmarks

TaskDatasetResultRank
Text-to-Audio GenerationAudioCaps (test)
KL Divergence1.86
195
Speech SynthesisLibriTTS (test)--
17
Music GenerationMusicCaps (test)
FAD1.37
16
Mixed-Audio GenerationMECAT Speech + Audio (S0A)
FADVGG2.17
10
Mixed-Audio GenerationMECAT 0MA Music + Audio
FADVGG Score3.25
5
Audio GenerationMECAT 00A
FADVGG4.07
5
Audio GenerationMECAT 0M0
FADVGG1.68
5
Audio GenerationMECAT S00
FADVGG1.76
5
Mixed-Audio GenerationMECAT Speech + Music (SM0)
FADVGG1.7
5
Showing 9 of 9 rows

Other info

Follow for update