Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Mind-Omni: A Unified Multi-Task Framework for Brain-Vision-Language Modeling via Discrete Diffusion

About

Modeling the interplay between external stimuli and internal neural representations is a pivotal research area for Brain-Computer Interfaces (BCIs). A major limitation of prior work is the prevailing paradigm of specialized, single-task models, which curtails versatility and neglects inter-task synergies. To address this, we propose Mind-Omni, the first versatile framework that unifies seven distinct encoding and decoding tasks through a discrete diffusion paradigm. At its core is a novel Brain Tokenizer that transforms heterogeneous, continuous brain signals into standardized, discrete tokens. This enables direct, token-level interactions for mutual understanding and generation between any two or more modalities within a shared semantic space. To unlock advanced reasoning capabilities, we further curate a specialized Brain Question Answering (BQA) instruction-tuning dataset. Our model not only establishes a new state-of-the-art among multi-task unified frameworks but also provides strong evidence for multi-task synergy. By demonstrating performance competitive with, and at times superior to, larger specialized models, our work offers a powerful new paradigm for neural modeling and paves the way for foundation models of neural activity. The code is publicly available at https://github.com/ReedOnePeck/Mind-Omni.

Yizhuo Lu, Changde Du, Qingyu Shi, Hang Chen, Jie Peng, Liuyun Jiang, Shuangchen Zhao, Huiguang He• 2026

Related benchmarks

TaskDatasetResultRank
Visual Reconstruction (B→I)NSD (Natural Scenes Dataset) (test)
AlexNet (k=2) Score67.1
6
Detailed DescriptionNatural Scenes Dataset (NSD) (test)
BLEU-129.12
6
Visual EncodingNatural Scenes Dataset (NSD) (test)
gPCC16
5
Brain Question AnsweringNSD (Natural Scenes Dataset) (test)
BLEU-123.18
3
Visual Reconstruction (B→I&T)NSD (Natural Scenes Dataset) (test)
PixCorr5.8
1
Showing 5 of 5 rows

Other info

Follow for update