Venice-H1: Failure-Aware Query Re-Ranking with Multi-Scale Grid Signatures for Referring Image Segmentation
About
Modern Referring Image Segmentation (RIS) systems generate multiple candidate masks per expression but rely on a simple heuristic--typically the argmax detection score--to select the final output. We identify query selection as a failure-case bottleneck: although heuristic selection succeeds on 82-93% of samples, the residual 7-18% of failures dominate the error budget, leaving a best-query selection gap of 3-11% mIoU. We introduce Venice-H1, a lightweight, backbone-decoupled post-hoc re-ranking module that encodes each candidate through multi-scale grid signatures--compact spatial descriptors pooled onto 4x4, 8x8, and 16x16 grids--and feeds them to a Transformer-based re-ranker with a Failure Gate (ROCAUC 0.78-0.82) that intervenes only when the default choice is likely suboptimal. Instantiated on DeRIS-L and DeRIS-B, Venice-H1 achieves delta_fail of +1.40 and +0.89 mIoU with strictly positive 95% CIs on all 16/16 (split, backbone) pairs and harmful-switch rates below 0.53%. Zero-shot transfer to medical referring segmentation (MS-CXR, M3D-RefSeg-2D) yields +1.16 and +0.51 mIoU without RIS-backbone fine-tuning. The module adds approximately 11.3M parameters and under 1 ms latency.
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Referring Image Segmentation | RefCOCO (val) | mIoU85.46 | 283 | |
| Referring Image Segmentation | RefCOCO+ (test-B) | mIoU78.84 | 276 | |
| Referring Image Segmentation | RefCOCO (test A) | mIoU86.51 | 254 | |
| Referring Image Segmentation | RefCOCO+ (val) | mIoU81.12 | 203 | |
| Referring Image Segmentation | RefCOCO (test-B) | mIoU84.57 | 195 | |
| Referring Image Segmentation | RefCOCO+ (testA) | mIoU83.46 | 121 | |
| Referring Image Segmentation | RefCOCOg (test(U)) | mIoU81.14 | 63 | |
| Referring Image Segmentation | RefCOCOg (val (U)) | mIoU79.8 | 63 |