Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

SpecAware: A Spectral-Content Aware Foundation Model for Unifying Multi-Sensor Learning in Hyperspectral Remote Sensing Mapping

About

Hyperspectral imaging (HSI) is a critical technique for fine-grained land-use and land-cover (LULC) mapping. However, the inherent heterogeneity of HSI data, particularly the variation in spectral channels across sensors, has long constrained the development of model generalization via transfer learning or joint training. Existing HSI foundation models show promise for different downstream tasks, but typically underutilize the critical guiding role of sensor meta-attributes and image semantic features, resulting in limited adaptability to cross-sensor joint learning. To address these issues, we propose SpecAware, which is a novel hyperspectral spectral-content aware foundation model for unifying multi-sensor learning for HSI mapping. To support this work, we constructed the Hyper-400K dataset, which is a new large-scale pre-training dataset with over 400\,k high-quality patches from diverse airborne AVIRIS sensors that cover two data processing levels (L1 and L2). The core of SpecAware is a hypernetwork-driven unified image embedding process for HSI data. Firstly, we designed a meta-content aware module to generate a unique conditional input for each HSI sample, tailored to each spectral band by fusing the sensor meta-attributes and its own image content. Secondly, we designed the HyperEmbedding module, where a sample-conditioned hypernetwork dynamically generates a pair of matrix factors for channel-wise encoding. This process implements two-step matrix factorization, consisting of adaptive spatial pattern extraction and latent semantic feature projection, yielding a unified hyperspectral token representation. Thus, SpecAware learns to capture and interpret spatial-spectral features across diverse scenes and sensors, enabling adaptive processing of variable spectral channels within a unified multi-sensor joint pre-training framework.

Renjie Ji, Xue Wang, Chao Niu, Wen Zhang, Yong Mei, Kun Tan• 2025

Related benchmarks

TaskDatasetResultRank
Scene ClassificationHRSSC (test)
OA85.22
11
Semantic segmentationEnMAP CDL
mIoU51.9
11
Semantic segmentationEnMAP BD-Foret
mIoU43.5
11
Semantic segmentationEnMAP BNTD
mIoU33.3
11
Semantic segmentationEnMAP EuCrops
mIoU44.2
11
Semantic segmentationEnMAP NLCD
mIoU31.7
11
Semantic segmentationEnMAP TreeMap
mIoU29.6
11
Semantic segmentationDESIS-CDL
mIoU46.8
11
Semantic segmentationEO-1 Hyperion CDL
mIoU50.8
11
Semantic segmentationGF-5 Wuhan
mIoU36.1
11
Showing 10 of 18 rows

Other info

Follow for update