EmoTaG: Emotion-Aware Talking Head Synthesis on Gaussian Splatting with Few-Shot Personalization
About
Audio-driven 3D talking head synthesis has advanced rapidly with Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS). By leveraging rich pre-trained priors, few-shot methods enable instant personalization from just a few seconds of video. However, under expressive facial motion, existing few-shot approaches often suffer from geometric instability and audio-emotion mismatch, highlighting the need for more effective emotion-aware motion modeling. In this work, we present EmoTaG, a few-shot emotion-aware 3D talking head synthesis framework built on the Pretrain-and-Adapt paradigm. Our key insight is to reformulate motion prediction in a structured FLAME parameter space rather than directly deforming 3D Gaussians, thereby introducing explicit geometric priors that improve motion stability. Building upon this, we propose a Gated Residual Motion Network (GRMN), which captures emotional prosody from audio while supplementing head pose and upper-face cues absent from audio, enabling expressive and coherent motion generation. Extensive experiments demonstrate that EmoTaG achieves state-of-the-art performance in emotional expressiveness, lip synchronization, visual realism, and motion stability.
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Talking Head Generation | MEAD self-driven | FID27.95 | 10 | |
| Audio-Video Synchronization | Cross-driven (Audio I) | Sync Error5.176 | 10 | |
| Audio-Video Synchronization | Cross-driven (Audio III) | Sync Error5.388 | 10 | |
| Audio-Video Synchronization | Cross-driven Audio II | Sync5.104 | 10 | |
| Audio-Video Synchronization | Cross-driven (Audio IV) | Sync Error5.317 | 10 | |
| Self-reconstruction | Neutral Set talking videos 5s (train) | PSNR30.02 | 7 | |
| Self-reconstruction | Emotional Set talking videos 5s (train) | PSNR29.95 | 7 | |
| Audio-visual talking head generation | User Study Cross-driven setting | Emotional Accuracy35 | 6 | |
| 3D talking head generation | Emotional self-reconstruction (test) | Emotional Expressiveness4.5 | 5 | |
| Lip synchronization | OOD audio-driven Cross Identity | Sync-E Score9.133 | 5 |