SOTA Language Model Evaluation on ArxivRollBench 2026a and PapersWithCode

70.8Valid Accuracy

moonshotai/kimi-k2.6

Updated 8d ago

Evaluation Results

Method
moonshotai/kimi-k2.6 2025.07	70.8	5.6	7.9
gemini-3.5-flash 2025.07	55.2	55.2	100
gpt-5.5 2025.07	54.4	54.4	100
claude-opus-4-7 2025.07	53.4	52.9	99
deepseek/deepseek-v4-pro 2025.07	53.3	25.4	47.7
x-ai/grok-4.3 2025.07	52	51.8	99.7
claude-sonnet-4-6 2025.07	48.3	48.3	100
moonshotai/kimi-k2.5 2025.07	46	12.3	26.8
deepseek/deepseek-v4-flash 2025.07	45.3	28	61.8
deepseek/deepseek-r1-0528 2025.07	43.2	14.3	33
claude-opus-4-6 2025.07	42.9	26.6	61.9
gpt-5.3-codex 2025.07	42.1	40	95
claude-haiku-4-5-20251001 2025.07	40.8	40.2	98.7
gpt-5.4 2025.07	39.3	39.3	100
Qwen/Qwen3-14B 2025.07	38.5	7.4	19.1
deepseek/deepseek-chat-v3-0324 2025.07	37.6	37.6	100
gpt-5.4-mini 2025.07	35.5	35.4	99.8
x-ai/grok-4.20 2025.07	34.5	34.5	100
deepseek/deepseek-chat-v3.1 2025.07	33.9	33.9	100
deepseek/deepseek-v3.2 2025.07	31.8	31.8	100
Qwen/Qwen2.5-7B-Instruct 2025.07	30	30	100
Qwen/Qwen2-7B-Instruct 2025.07	29.5	29.4	99.8
Qwen/Qwen3-8B 2025.07	29.3	29.3	99.8
Qwen/Qwen2.5-7B 2025.07	28.2	28.2	100
meta-llama/Llama-3.1-8B-Instruct 2025.07	27.7	27.7	100
Qwen/Qwen3-4B 2025.07	27.4	19.7	71.8
mistralai/Mistral-7B-Instruct-v0.3 2025.07	26.2	26.2	100
mistralai/Mistral-7B-Instruct-v0.1 2025.07	25.8	25.8	100
tiiuae/Falcon3-10B-Instruct 2025.07	23.8	23.8	99.9
mistralai/Mistral-7B-Instruct-v0.2 2025.07	23.7	22.7	95.6
meta-llama/Llama-3.2-3B 2025.07	23.4	23.4	100
meta-llama/Llama-3.2-3B-Instruct 2025.07	23.2	23.2	100
meta-llama/Llama-3.2-1B 2025.07	22.1	22.1	100
meta-llama/Llama-3.1-8B 2025.07	21.9	21.9	100
Qwen/Qwen2.5-Math-7B 2025.07	21.6	21.2	98.2
microsoft/Phi-4-reasoning-plus 2025.07	21.2	20.3	95.6
meta-llama/Llama-3.2-1B-Instruct 2025.07	12.6	12.6	100
microsoft/Phi-4-reasoning 2025.07	8.3	4.8	58.5
Qwen/Qwen2.5-Math-7B-Instruct 2025.07	7.5	7.5	100
deepseek/deepseek-v3.2-speciale 2025.07	0	0	0
gemini-3-flash-preview 2025.07	0	0	100
gemini-3.1-pro-preview 2025.07	0	0	100