VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perception
About
Inducing reasoning in multimodal large language models (MLLMs) is critical for achieving human-level perception and understanding. Existing methods mainly leverage LLM reasoning to analyze parsed visuals, often limited by static perception stages. This paper introduces Visual Test-Time Scaling (VTTS), a novel approach to enhance MLLMs' reasoning via iterative perception during inference. VTTS mimics humans' hierarchical attention by progressively refining focus on high-confidence spatio-temporal regions, guided by updated textual predictions. Specifically, VTTS employs an Iterative Perception (ITP) mechanism, incorporating reinforcement learning with spatio-temporal supervision to optimize reasoning. To support this paradigm, we also present VTTS-80K, a dataset tailored for iterative perception. These designs allows a MLLM to enhance its performance by increasing its perceptual compute. Extensive experiments validate VTTS's effectiveness and generalization across diverse tasks and benchmarks. Our newly introduced Videochat-R1.5 model has achieved remarkable improvements, with an average increase of over 5\%, compared to robust baselines such as Qwen2.5VL-3B and -7B, across more than 15 benchmarks that encompass video conversation, video reasoning, and spatio-temporal perception.
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Video Question Answering | VideoMME | Accuracy65.2 | 99 | |
| Video Question Answering | VideoMMMU | Accuracy49.6 | 52 | |
| Temporal Action Localization | ActivityNet v1.3 (test) | -- | 47 | |
| Temporal Grounding | ActivityNet Captions | Recall@1 (IoU=0.5)15.6 | 45 | |
| Video Question Answering | LongVideoBench | Accuracy61.4 | 34 | |
| Temporal Grounding | Charades-STA | mIoU60.6 | 33 | |
| Video Reasoning | SAGE-Bench 1.0 (test) | Overall Score54.8 | 29 | |
| Grounded Video Question Answering | NExT-GQA (test) | mIoU20.5 | 24 | |
| Temporal Video Grounding | QVHighlights TimeLens (test) | Recall @ IoU=0.362.2 | 17 | |
| Temporal Video Grounding | ActivityNet TimeLens (test) | Recall@0.340.6 | 17 |