Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

LIMSSR: LLM-Driven Sequence-to-Score Reasoning under Training-Time Incomplete Multimodal Observations

About

Real-world multimodal learning is often hindered by missing modalities. While Incomplete Multimodal Learning (IML) has gained traction, existing methods typically rely on the unrealistic assumption of full-modal availability during training to provide reconstruction supervision or cross-modal priors. This paper tackles the more challenging setting of IML under training-time incomplete observations, which precludes reliance on a ``God's eye view'' of complete data. We propose LIMSSR (LLM-Driven Incomplete Multimodal Sequence-to-Score Reasoning), a framework that reformulates this challenge as a conditional sequence reasoning task. LIMSSR leverages the semantic reasoning capabilities of Large Language Models via Prompt-Guided Context-Aware Modality Imputation and Multidimensional Representation Fusion to infer latent semantics from available contexts without direct reconstruction. To mitigate hallucinations, we introduce a Mask-Aware Dual-Path Aggregation to dynamically calibrate inference uncertainty. Extensive experiments on three Action Quality Assessment datasets demonstrate that LIMSSR significantly outperforms state-of-the-art baselines without relying on complete training data, establishing a new paradigm for data-efficient multimodal learning. Code is available at https://github.com/XuHuangbiao/LIMSSR.

Huangbiao Xu, Huanqi Wu, Xiao Ke, Yuxin Peng• 2026

Related benchmarks

TaskDatasetResultRank
Action Quality AssessmentFis-V
TES Spearman Correlation0.792
22
Action Quality AssessmentFS1000 7-class
Spearman Correlation ({v, f})0.854
9
Action Quality AssessmentFis-V 2-class
Spearman Correlation ({v, f})0.824
9
Action Quality AssessmentRG 4-class
Spearman Correlation ({v, f})0.825
9
Sequence-to-score reasoningFS1000 1.0 (test)
Average SRCC0.789
9
Action Quality AssessmentFS1000
TES Spearman Correlation0.907
8
Action Quality AssessmentRG
Spearman Correlation (Ball)0.813
8
Showing 7 of 7 rows

Other info

Follow for update