Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

TASU2: Controllable CTC Simulation for Alignment and Low-Resource Adaptation of Speech LLMs

About

Speech LLM post-training increasingly relies on efficient cross-modal alignment and robust low-resource adaptation, yet collecting large-scale audio-text pairs remains costly. Text-only alignment methods such as TASU reduce this burden by simulating CTC posteriors from transcripts, but they provide limited control over uncertainty and error rate, making curriculum design largely heuristic. We propose \textbf{TASU2}, a controllable CTC simulation framework that simulates CTC posterior distributions under a specified WER range, producing text-derived supervision that better matches the acoustic decoding interface. This enables principled post-training curricula that smoothly vary supervision difficulty without TTS. Across multiple source-to-target adaptation settings, TASU2 improves in-domain and out-of-domain recognition over TASU, and consistently outperforms strong baselines including text-only fine-tuning and TTS-based augmentation, while mitigating source-domain performance degradation.

Jing Peng, Chenghao Wang, Yi Yang, Lirong Qian, Junjie Li, Yu Xi, Shuai Wang, Kai Yu• 2026

Related benchmarks

TaskDatasetResultRank
Automatic Speech RecognitionLibriSpeech Other
WER8.15
96
Automatic Speech RecognitionLibriSpeech Clean
WER3.41
80
Automatic Speech RecognitionTED-LIUM 3
WER13.93
45
Automatic Speech RecognitionSlideSpeech
WER16.41
6
Speech-to-text TranslationCoVoST2 en-zh
BLEU33.08
5
Showing 5 of 5 rows

Other info

Follow for update