Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Mamba-3: Improved Sequence Modeling using State Space Principles

About

Scaling inference-time compute has emerged as an important driver of LLM performance, making inference efficiency a central focus of model design alongside model quality. While the current Transformer-based models deliver strong model quality, their quadratic compute and linear memory make inference expensive. This has spurred the development of sub-quadratic models with reduced linear compute and constant memory requirements. However, many recent linear models trade off model quality and capability for algorithmic efficiency, failing on tasks such as state tracking. Moreover, their theoretically linear inference remains hardware-inefficient in practice. Guided by an inference-first perspective, we introduce three core methodological improvements inspired by the state space model (SSM) viewpoint of linear models. We combine: (1) a more expressive recurrence derived from SSM discretization, (2) a complex-valued state update rule that enables richer state tracking, and (3) a multi-input, multi-output (MIMO) formulation for better model performance without increasing decode latency. Together with architectural refinements, our Mamba-3 model achieves significant gains across retrieval, state-tracking, and downstream language modeling tasks. At the 1.5B scale, Mamba-3 improves average downstream accuracy by 0.6 percentage points compared to the next best model (Gated DeltaNet), with Mamba-3's MIMO variant further improving accuracy by another 1.2 points for a total 1.8 point gain. Across state-size experiments, Mamba-3 achieves comparable perplexity to Mamba-2 despite using half of its predecessor's state size. Our evaluations demonstrate Mamba-3's ability to advance the performance-efficiency Pareto frontier.

Aakash Lahoti, Kevin Y. Li, Berlin Chen, Caitlin Wang, Aviv Bick, J. Zico Kolter, Tri Dao, Albert Gu• 2026

Related benchmarks

TaskDatasetResultRank
Commonsense ReasoningWinoGrande--
1442
Question AnsweringARC Challenge--
906
Commonsense ReasoningPIQA
Accuracy75.3
757
Commonsense ReasoningHellaSwag
HellaSwag Accuracy62.3
711
Question AnsweringARC Easy
Accuracy76.5
597
Language ModelingLAMBADA
Accuracy51.7
412
Sentence CompletionHellaSwag
Accuracy26
364
Language ModelingWikiText
Word Perplexity15.54
234
Word PredictionLAMBADA
Accuracy34
192
Language ModelingFineWeb-Edu
PPL10.24
141
Showing 10 of 28 rows

Other info

Follow for update