Multimodal Group Emotion Recognition In-the-Wild Towards a Privacy-Safe Non-Individual Approach
About
This thesis addresses group emotion recognition (GER) in-the-wild with a focus on privacy preservation. Unlike traditional emotion recognition methods that rely on individual-level cues such as face, gaze, or voice analysis, this work uses collective audio-video signals to infer emotions at the group level, reducing risks of individual monitoring and surveillance. Two complementary frameworks are proposed. The first is a cross-attention multimodal architecture for audio-video fusion, combined with Frames Attention Pooling (FAP) for temporal aggregation. It is supported by synthetic data augmentation and validated through ablation studies, demonstrating robustness in real-world GER conditions. The second framework, Variational Encoder Multi-Decoder (VE-MD), learns a shared latent space for emotion classification and structural representation prediction, including body and face cues. Two decoding strategies, DETR-based and heatmap-based, are explored to analyze the role of structural representations in group and individual settings. The thesis makes three main contributions: it clarifies the role of multimodality and structural cues in group-level affective computing; introduces two architectures for privacy-preserving multimodal GER; and shows that competitive performance can be achieved without using individual features as input data.
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Facial Expression Recognition | RAF-DB | -- | 99 | |
| Facial Expression Recognition | CK+ | -- | 83 | |
| Facial Expression Recognition | AffectNet 7 classes | -- | 63 | |
| Dynamic Facial Expression Recognition | DFEW | WAR70.73 | 55 | |
| Emotion Recognition | CREMA-D | -- | 54 | |
| Emotion Recognition | RAVDESS | -- | 46 | |
| Speech Emotion Recognition | RAVDESS | -- | 43 | |
| Emotion Recognition | DEAP | -- | 36 | |
| Engagement Recognition | EngageNet | Accuracy68.98 | 26 | |
| Group Emotion Recognition | VGAF | Accuracy82.25 | 19 |