Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

FedMAC: Tackling Partial-Modality Missing in Federated Learning with Cross-Modal Aggregation and Contrastive Regularization

About

Federated Learning (FL) is a method for training machine learning models using distributed data sources. It ensures privacy by allowing clients to collaboratively learn a shared global model while storing their data locally. However, a significant challenge arises when dealing with missing modalities in clients' datasets, where certain features or modalities are unavailable or incomplete, leading to heterogeneous data distribution. While previous studies have addressed the issue of complete-modality missing, they fail to tackle partial-modality missing on account of severe heterogeneity among clients at an instance level, where the pattern of missing data can vary significantly from one sample to another. To tackle this challenge, this study proposes a novel framework named FedMAC, designed to address multi-modality missing under conditions of partial-modality missing in FL. Additionally, to avoid trivial aggregation of multi-modal features, we introduce contrastive-based regularization to impose additional constraints on the latent representation space. The experimental results demonstrate the effectiveness of FedMAC across various client configurations with statistical heterogeneity, outperforming baseline methods by up to 26% in severe missing scenarios, highlighting its potential as a solution for the challenge of partially missing modalities in federated systems. Our source code is provided at https://github.com/nmduonggg/PEPSY

Manh Duong Nguyen, Trung Thanh Nguyen, Huy Hieu Pham, Trong Nghia Hoang, Phi Le Nguyen, Thanh Trung Huynh• 2024

Related benchmarks

TaskDatasetResultRank
Multimodal Sentiment AnalysisMOSEI
MAE0.68
183
Multimodal Sentiment AnalysisMOSI
MAE0.83
132
Node ClassificationGrocery
Accuracy83.47
71
Node ClassificationEle-fashion
Accuracy81.28
26
Modality RetrievalToys
R@50.5978
11
Modality RetrievalFlickr30K
Recall@563.22
11
Link PredictionDY
AUC84.48
10
Link PredictionBili_Dance
AUC79.37
10
Showing 8 of 8 rows

Other info

Follow for update