Share your thoughts, 1 month free Claude Pro on usSee more

Entity Counting on NL Counting (held-out)

100Accuracy

Oracle

Updated 2mo ago

Evaluation Results

Method	Links
Oracle 2026.05		100
DPS (hard, multi-seed protocol) 2026.05		98.7
Full lm_head (held-out) 2026.05		94.2
9-row lm_head (held-out) 2026.05		93.8
9-row lm_head (held-out) 2026.05		92
LoRA Q/V rank-16 (fullvocab) 2026.05		91.7
9-row lm_head + DPS (matched train, 3 seeds) 2026.05		90.3
LoRA (all 36 layers, constrained-gen) 2026.05		84
DPS layer sweep (10 layers, best) 2026.05		28.7
DPS (α ∈ [5, 500], best) 2026.05		26.5
Baseline (vanilla) 2026.05		26.2
Baseline (vanilla) 2026.05		24.5
Shuffled-row control 2026.05		12
Baseline (vanilla) 2026.05		11.3