Nano-EmoX: Unifying Multimodal Emotional Intelligence from Perception to Empathy
About
The development of affective multimodal language models (MLMs) has long been constrained by a gap between low-level perception and high-level interaction, leading to fragmented affective capabilities and limited generalization. To bridge this gap, we propose a cognitively inspired three-level hierarchy that organizes affective tasks according to their cognitive depth-perception, understanding, and interaction-and provides a unified conceptual foundation for advancing affective modeling. Guided by this hierarchy, we introduce Nano-EmoX, a small-scale multitask MLM, and P2E (Perception-to-Empathy), a curriculum-based training framework. Nano-EmoX integrates a suite of omni-modal encoders, including an enhanced facial encoder and a fusion encoder, to capture key multimodal affective cues and improve cross-task transferability. The outputs are projected into a unified language space via heterogeneous adapters, empowering a lightweight language model to tackle diverse affective tasks. Concurrently, P2E progressively cultivates emotional intelligence by aligning rapid perception with chain-of-thought-driven empathy. To the best of our knowledge, Nano-EmoX is the first compact MLM (2.2B) to unify six core affective tasks across all three hierarchy levels, achieving state-of-the-art or highly competitive performance across multiple benchmarks, demonstrating excellent efficiency and generalization. The code is available at https://github.com/waHAHJIAHAO/Nano-EmoX.
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Multimodal Intent Recognition | MIntRec | Accuracy58.2 | 19 | |
| MSA, MER and fine-grained OV-MER tasks | MER2023, MER2024, MELD, IEMOCAP, MOSI, MOSEI, SIMS, SIMSV2, OV-MERD | Accuracy (MER2023)79.09 | 12 | |
| Emotion Recognition | ERI (test) | Clue Overlap7.83 | 12 | |
| Multimodal Intent Recognition | MIntRec 2.0 | Accuracy47.32 | 10 | |
| Empathetic Response Generation | ERG | Distinct-1 Score95.47 | 9 |