Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Graph Signal Processing Meets Mamba2: Adaptive Filter Bank via Delta Modulation

About

State-space models (SSMs) offer efficient alternatives to attention with linear-time recurrence. Mamba2, a recent SSM-based language model, uses selective input gating and a multi-head structure, enabling parallel computation and strong benchmark performance. However, its multi-head recurrence operates independently without structured utilization or analysis. In this work, we propose a novel method called Hierarchical ADaptive filter bank for Efficient SSMs (HADES), a Graph Signal Processing (GSP)-inspired framework that reinterprets Mamba2 as an adaptive filter bank on a line graph. Our hierarchical architecture introduces two filter types: shared filters for global low-pass behavior and expert filters for local high-pass behavior, achieved through structured bias on the parameter {\Delta}. HADES achieves comparable performance to baseline models including Mamba2 across various benchmarks in language modeling, commonsense reasoning, and long-context retrieval, while using only 58.9% of the original parameters. In this regard, HADES bridges GSP and neural sequence modeling, enabling efficient, hierarchical, and interpretable filtering within state-space models.

Yehjin Shin, Seojin Kim, Noseong Park• 2026

Related benchmarks

TaskDatasetResultRank
Commonsense ReasoningHellaSwag--
1896
Commonsense ReasoningWinoGrande
Accuracy56.35
1442
Commonsense ReasoningPIQA
Accuracy71.33
757
Language ModelingWikiText
PPL20.41
740
Commonsense ReasoningHellaSwag
HellaSwag Accuracy51.85
711
Language ModelingLAMBADA
Accuracy41.18
412
Commonsense ReasoningARC Challenge
Accuracy34.81
243
Common Sense ReasoningBoolQ
Accuracy60.73
240
Language ModelingLAMBADA
Perplexity21.74
198
Commonsense ReasoningOpenBookQA
Accuracy38.6
108
Showing 10 of 16 rows

Other info

Follow for update