Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Efficient Low-Resource Language Adaptation via Multi-Source Dynamic Logit Fusion

About

Adapting large language models (LLMs) to low-resource languages (LRLs) is constrained by the scarcity of task data and computational resources. Although Proxy Tuning offers a logit-level strategy for introducing scaling effects, it often fails in LRL settings because the large model's weak LRL competence might overwhelm the knowledge of specialized smaller models. We thus propose TriMix, a test-time logit fusion framework that dynamically balances capabilities from three different sources: LRL competence from a continually pretrained small model, task competence from high-resource language instruction tuning, and the scaling benefits of large models. It is data- and compute-efficient, requiring no LRL task annotations, and only continual pretraining on a small model. Experiments across four model families and eight LRLs show that TriMix consistently outperforms single-model baselines and Proxy Tuning. Our analysis reveals that prioritizing the small LRL-specialized model's logits is crucial for success, challenging the prevalent large-model-dominant assumption.

Chen Zhang, Jiuheng Lin, Zhiyuan Liao, Yansong Feng• 2026

Related benchmarks

TaskDatasetResultRank
Multilingual Language UnderstandingQwen Multi-task Evaluation Suite 2.5 (test)
MC Score59.5
18
Intent ClassificationSIB-200 Tamil
Accuracy56.6
5
Intent ClassificationSIB-200 Telugu
Accuracy71.7
5
Intent ClassificationSIB-200 Odia
Accuracy60.6
5
Intent ClassificationSIB-200 Bengali
Accuracy74.7
5
Low-resource language evaluationMiLiC-Eval
BOD54.8
5
Low-resource language evaluationMiLiC-Eval
BOD Score30.4
5
Reading ComprehensionBelebele Telugu
Accuracy29.9
5
Reading ComprehensionBelebele Odia
Accuracy25.7
5
Reading ComprehensionBelebele Tamil
Accuracy32.6
5
Showing 10 of 12 rows

Other info

Follow for update