Our new X account is live! Follow @wizwand_team for updates
WorkDL logo mark

ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attention

About

Modern multimodal large language models (MLLMs) adopt a unified self-attention design that processes visual and textual tokens at every Transformer layer, incurring substantial computational overhead. In this work, we revisit the necessity of such dense visual processing and show that projected visual embeddings are already well-aligned with the language space, while effective vision-language interaction occurs in only a small subset of layers. Based on these insights, we propose ViCA (Vision-only Cross-Attention), a minimal MLLM architecture in which visual tokens bypass all self-attention and feed-forward layers, interacting with text solely through sparse cross-attention at selected layers. Extensive evaluations across three MLLM backbones, nine multimodal benchmarks, and 26 pruning-based baselines show that ViCA preserves 98% of baseline accuracy while reducing visual-side computation to 4%, consistently achieving superior performance-efficiency trade-offs. Moreover, ViCA provides a regular, hardware-friendly inference pipeline that yields over 3.5x speedup in single-batch inference and over 10x speedup in multi-batch inference, reducing visual grounding to near-zero overhead compared with text-only LLMs. It is also orthogonal to token pruning methods and can be seamlessly combined for further efficiency gains. Our code is available at https://github.com/EIT-NLP/ViCA.

Wenjie Liu, Hao Wu, Xin Qiu, Yingqi Fan, Yihan Zhang, Anhao Zhao, Yunpu Ma, Xiaoyu Shen• 2026

Related benchmarks

TaskDatasetResultRank
Visual Question AnsweringVQA v2
Accuracy76.6
1165
Object Hallucination EvaluationPOPE
Accuracy86.7
935
Text-based Visual Question AnsweringTextVQA
Accuracy55.5
496
Visual Question AnsweringGQA
Accuracy60.4
374
Multimodal UnderstandingMMBench--
367
Multimodal UnderstandingMMBench CN
Accuracy57.7
162
Science Question AnsweringScienceQA SQA-IMG
Accuracy69.3
114
Multimodal UnderstandingMMBench (MMB)
Accuracy64
69
Multimodal PerceptionMME Perception
Perception Score1.46e+3
61
Multimodal UnderstandingSEED-I Image
Accuracy0.632
40
Showing 10 of 12 rows

Other info

Follow for update