Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Native Active Perception as Reasoning for Omni-Modal Understanding

About

Passive models for long video understanding typically rely on a "watch-it-all" paradigm, processing frames uniformly regardless of query difficulty, causing computational cost to grow with video duration. Although interactive frameworks have emerged, they often rely on global pre-scanning, and their context cost still scales with video length. We propose OmniAgent, the first native omni-modal agent that formulates video understanding as a POMDP-based iterative Observation-Thought-Action cycle. OmniAgent executes on-demand actions to selectively distill audio-visual cues into a persistent textual memory, effectively decoupling reasoning complexity from raw video duration. To operationalize this, we introduce (1) Agentic Supervised Fine-Tuning to bootstrap native active perception via best-of-N trajectory synthesis with dual-stage quality control, and (2) Agentic Reinforcement Learning with TAURA (Turn-aware Adaptive Uncertainty Rescaled Advantage), which leverages turn-level entropy to steer credit assignment toward pivotal discovery turns. Crucially, OmniAgent exhibits positive test-time scaling, where performance improves as the number of reasoning turns increases, validating the efficacy of active perception. Empirical results across ten benchmarks (e.g., VideoMME, LVBench) demonstrate that OmniAgent achieves state-of-the-art performance among open-source models. Notably, on LVBench, our 7B agent outperforms the 10$\times$ larger Qwen2.5-VL-72B (50.5% vs. 47.3%).

Zhenghao Xing, Ruiyang Xu, Yuxuan Wang, Jinzheng He, Ziyang Ma, Qize Yang, Yunfei Chu, Jin Xu, Junyang Lin, Chi-Wing Fu, Pheng-Ann Heng• 2026

Related benchmarks

TaskDatasetResultRank
Long Video UnderstandingLVBench--
267
Long Video UnderstandingMLVU--
265
Video ReasoningVSI-Bench
Accuracy48.4
101
Audiovisual Understanding & ReasoningDaily-Omni
Score64.8
33
Video UnderstandingVideoMME w/o sub. Overall
Overall Accuracy67.8
22
Video UnderstandingVideoMME w/o sub. Long
Long Accuracy59.6
16
Video Reasoning and Long Video UnderstandingMinerva
AVG Score41.4
9
Audio-visual temporal groundingVUE (TR)
Vision Score46.1
7
Audio-Visual Understanding and ReasoningWorldSense
AVG47.2
6
Audio-Visual Understanding and ReasoningOmniVideo
Average Score37.1
5
Showing 10 of 11 rows

Other info

GitHub

Follow for update