Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Venice-H1: Failure-Aware Query Re-Ranking with Multi-Scale Grid Signatures for Referring Image Segmentation

About

Modern Referring Image Segmentation (RIS) systems generate multiple candidate masks per expression but rely on a simple heuristic--typically the argmax detection score--to select the final output. We identify query selection as a failure-case bottleneck: although heuristic selection succeeds on 82-93% of samples, the residual 7-18% of failures dominate the error budget, leaving a best-query selection gap of 3-11% mIoU. We introduce Venice-H1, a lightweight, backbone-decoupled post-hoc re-ranking module that encodes each candidate through multi-scale grid signatures--compact spatial descriptors pooled onto 4x4, 8x8, and 16x16 grids--and feeds them to a Transformer-based re-ranker with a Failure Gate (ROCAUC 0.78-0.82) that intervenes only when the default choice is likely suboptimal. Instantiated on DeRIS-L and DeRIS-B, Venice-H1 achieves delta_fail of +1.40 and +0.89 mIoU with strictly positive 95% CIs on all 16/16 (split, backbone) pairs and harmful-switch rates below 0.53%. Zero-shot transfer to medical referring segmentation (MS-CXR, M3D-RefSeg-2D) yields +1.16 and +0.51 mIoU without RIS-backbone fine-tuning. The module adds approximately 11.3M parameters and under 1 ms latency.

Nicol\`o Savioli• 2026

Related benchmarks

TaskDatasetResultRank
Referring Image SegmentationRefCOCO (val)
mIoU85.46
283
Referring Image SegmentationRefCOCO+ (test-B)
mIoU78.84
276
Referring Image SegmentationRefCOCO (test A)
mIoU86.51
254
Referring Image SegmentationRefCOCO+ (val)
mIoU81.12
203
Referring Image SegmentationRefCOCO (test-B)
mIoU84.57
195
Referring Image SegmentationRefCOCO+ (testA)
mIoU83.46
121
Referring Image SegmentationRefCOCOg (test(U))
mIoU81.14
63
Referring Image SegmentationRefCOCOg (val (U))
mIoU79.8
63
Showing 8 of 8 rows

Other info

Follow for update