Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Graph Signal Processing Meets Mamba2: Adaptive Filter Bank via Delta Modulation

About

State-space models (SSMs) offer efficient alternatives to attention with linear-time recurrence. Mamba2, a recent SSM-based language model, uses selective input gating and a multi-head structure, enabling parallel computation and strong benchmark performance. However, its multi-head recurrence operates independently without structured utilization or analysis. In this work, we propose a novel method called Hierarchical ADaptive filter bank for Efficient SSMs (HADES), a Graph Signal Processing (GSP)-inspired framework that reinterprets Mamba2 as an adaptive filter bank on a line graph. Our hierarchical architecture introduces two filter types: shared filters for global low-pass behavior and expert filters for local high-pass behavior, achieved through structured bias on the parameter {\Delta}. HADES achieves comparable performance to baseline models including Mamba2 across various benchmarks in language modeling, commonsense reasoning, and long-context retrieval, while using only 58.9% of the original parameters. In this regard, HADES bridges GSP and neural sequence modeling, enabling efficient, hierarchical, and interpretable filtering within state-space models.

Yehjin Shin, Seojin Kim, Noseong Park• 2026

Related benchmarks

TaskDatasetResultRank
Commonsense ReasoningHellaSwag--
1891
Commonsense ReasoningWinoGrande
Accuracy56.35
1085
Commonsense ReasoningPIQA
Accuracy71.33
751
Language ModelingWikiText
PPL20.41
732
Commonsense ReasoningHellaSwag
HellaSwag Accuracy51.85
350
Language ModelingLAMBADA
Accuracy41.18
268
Common Sense ReasoningBoolQ
Accuracy60.73
212
Commonsense ReasoningARC Challenge
Accuracy34.81
190
Language ModelingLAMBADA
Perplexity21.74
150
Common Sense ReasoningARC Easy
ARC (easy) Accuracy68.48
72
Showing 10 of 16 rows

Other info

Follow for update