Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation

About

Reasoning Video Object Segmentation (RVOS) demands a sophisticated integration of temporal dynamics, spatial details, and linguistic reasoning to achieve precise pixel-level localization. Existing methods are limited to reasoning over fixed initial inputs and lack the capacity to actively acquire further visual evidence, which is often essential for resolving complex references in long or intricate videos. To address this, we propose \textbf{VideoSEG-O3}, the first multi-turn reinforcement learning framework for RVOS that emulates the human \textit{``coarse-to-fine''} cognitive process. It employs a \textit{multi-turn temporal-spatial chain-of-thought} to capture fine-grained details by iteratively pinpointing critical intervals and keyframes. Additionally, to enable the policy to perceive segmentation quality beyond mere text probability of \texttt{[SEG]} during the RL stage, we introduce \textit{SEG-aware logit calibration}, which integrates pixel-wise segmentation feedback directly into the token-level logits. Furthermore, we design a \textit{decoupled thinking trace} to hierarchically decompose the reasoning process into temporal, spatial, and linguistic dimensions, and construct \textbf{VTS-CoT}, a specialized cold-start dataset featuring comprehensive reasoning trajectories. The code and models will be released at https://github.com/Dmmm1997/VideoSEG-O3.

Ming Dai, Sen Yang, Boqiang Duan, Boyuan Tong, Jiedong Zhuang, Wankou Yang, Jingdong Wang• 2026

Related benchmarks

TaskDatasetResultRank
Reasoning SegmentationReasonSeg (val)--
382
Reasoning SegmentationReasonSeg (test)--
287
Referring Image SegmentationRefCOCO (val)--
283
Referring Image SegmentationRefCOCO+ (test-B)--
276
Referring Image SegmentationRefCOCO (test A)--
254
Referring Video Object SegmentationRef-DAVIS 2017 (val)
J&F80
240
Referring Image SegmentationRefCOCO+ (val)--
203
Referring Image SegmentationRefCOCO (test-B)--
195
Referring Image SegmentationRefCOCO+ (testA)--
121
Referring Image SegmentationRefCOCOg (val (U))--
63
Showing 10 of 19 rows

Other info

Follow for update