Language Modeling on Nemotron-CC 25M held-out tokens

9.58Perplexity

MoE Transformer w/ dynamic conv. (low-rank)

Updated 1mo ago

Evaluation Results

Method	Links
MoE Transformer w/ dynamic conv. (low-rank) 2026.06		9.58
MoE Transformer w/ dynamic conv. (head-wise) 2026.06		9.65
MoE Transformer w/ static conv. 2026.06		9.74
MoE Transformer 2026.06		9.86
Transformer w/ dynamic conv. (low-rank) 2026.06		10.95
Transformer w/ dynamic conv. (low-rank) 2026.06		11.24
Transformer w/ dynamic conv. (head-wise) 2026.06		11.41
Transformer w/ static conv. 2026.06		11.5
Transformer w/ more params (wider FFN) 2026.06		11.67
Transformer 2026.06		11.71
Transformer w/ dynamic conv. (low-rank) 2026.06		17.42
Gated DeltaNet w/ dynamic conv. (low-rank) 2026.06		17.95
Transformer w/ dynamic conv. (low-rank) 2026.06		18.01
Gated DeltaNet w/ dynamic conv. (head-wise) 2026.06		18.03
Transformer w/ dynamic conv. (head-wise) 2026.06		18.22
Transformer w/ static conv. 2026.06		18.66
Mamba-2 w/ dynamic conv. (head-wise) 2026.06		18.69
Mamba-2 w/ dynamic conv. (low-rank) 2026.06		18.72
Gated DeltaNet w/ static conv. 2026.06		18.75
Gated DeltaNet (w/o conv.) 2026.06		18.93
Transformer w/ more params (wider FFN) 2026.06		18.99
Transformer 2026.06		19.12
Mamba-2 w/ static conv. 2026.06		19.3
Mamba-2 (w/o conv.) 2026.06		20.26