Hyperpixel Flow: Semantic Correspondence with Multi-layer Neural Features
About
Establishing visual correspondences under large intra-class variations requires analyzing images at different levels, from features linked to semantics and context to local patterns, while being invariant to instance-specific details. To tackle these challenges, we represent images by "hyperpixels" that leverage a small number of relevant features selected among early to late layers of a convolutional neural network. Taking advantage of the condensed features of hyperpixels, we develop an effective real-time matching algorithm based on Hough geometric voting. The proposed method, hyperpixel flow, sets a new state of the art on three standard benchmarks as well as a new dataset, SPair-71k, which contains a significantly larger number of image pairs than existing datasets, with more accurate and richer annotations for in-depth analysis.
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Semantic Correspondence | SPair-71k (test) | PCK@0.128.2 | 122 | |
| Semantic Correspondence | PF-WILLOW | PCK@0.1 (bbox)76.3 | 109 | |
| Semantic Correspondence | PF-Pascal (test) | PCK@0.184.8 | 106 | |
| Semantic Correspondence | PF-PASCAL | PCK @ alpha=0.184.8 | 98 | |
| Keypoint Transfer | SPair-71k (test) | Bicycle18.9 | 38 | |
| Semantic Correspondence | PF-WILLOW (test) | PCK @ 0.10 (bbox)74.4 | 37 | |
| Semantic keypoint transfer | PF-Pascal (test) | PCK @ 0.0563.5 | 35 | |
| Semantic Correspondence | Caltech-101 | LT-ACC88 | 31 | |
| Semantic Correspondence | PF-PASCAL | PCK@0.188.3 | 29 | |
| Semantic Matching | TSS (test) | FG3DCar PCK@0.0593.6 | 27 |