Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Multilingual Long-Form Speech Instruction Following: KIT's Submission to IWSLT 2026

About

With the advent of Large Language Models, single-task and token-based multi-task models have evolved into instruction-based systems that infer task and target language implicitly from natural language prompts. This trend is reflected in IWSLT's Instruction Following Track, which this year introduced new tasks including an unknown surprise task, posing a genuine challenge against overfitting to known tasks. We present KIT's submission to the Long and Short Instruction Following tracks in the unconstrained setting. Our approach combines a general data augmentation pipeline that converts short-form corpora into long-form training data through segment concatenation, LLM-based label generation, and cross-lingual translation, yielding over 1M instances across six tasks and four languages. We further show that likelihood-based re-ranking, while highly effective for ASR, systematically degrades semantic tasks by spuriously selecting candidates generated from segmented audio processing rather than holistic long-form inference, a failure mode resolved by combining likelihood with Minimum Bayes Risk decoding.

Enes Yavuz Ugan, Maike Z\"ufle, Yuka Ko, Supriti Sinhamahapatra, Fabian Retkowski, Seymanur Akti, Jan Niehues, Alexander Waibel• 2026

Related benchmarks

TaskDatasetResultRank
Speech TranslationMCIF long track
COMET Score83.75
20
Spoken Question AnsweringMCIF long track
BERTScore40.86
20
Spoken SummarizationMCIF long track
BERTScore29.74
20
Automatic Speech RecognitionMCIF long track
WER5.9
20
Speech Instruction-FollowingIWSLT Short Track
COMET Score0.844
2
Speech Instruction-FollowingIWSLT Long Track
ST (COMET)0.843
2
Showing 6 of 6 rows

Other info

Follow for update