SOTA Research Paper Reasoning and Comprehension on ArxivRollBench 2025a (val) and PapersWithCode

44.3Valid Accuracy

google/gemini-2.5-flash

Updated 8d ago

Evaluation Results

Method
google/gemini-2.5-flash 2025.07	44.3	44.3	100
anthropic/claude-3.5-sonnet 2025.07	43.1	43.1	100
anthropic/claude-3.7-sonnet 2025.07	42.9	42.9	100
anthropic/claude-opus-4 2025.07	42.7	42.7	100
anthropic/claude-sonnet-4 2025.07	41.2	41.2	100
deepseek/deepseek-chat-v3-0324 2025.07	40.5	40.5	100
google/gemini-2.0-flash-001 2025.07	39.5	39.5	100
moonshotai/kimi-k2 2025.07	37	37	100
openai/gpt-4o 2025.07	36.2	36.2	100
openai/gpt-4 2025.07	35.3	35.3	100
nvidia/Llama-3.1-Nemotron-70B-Instruct-HF 2025.07	33	33	100
meta-llama/Llama-3.1-70B-Instruct 2025.07	32.2	32.2	100
meta-llama/Llama-3.3-70B-Instruct 2025.07	30.1	30.1	100
Qwen/Qwen2.5-7B-Instruct 2025.07	29.6	29.6	100
Qwen/Qwen2-7B-Instruct 2025.07	29	29	100
Qwen/Qwen3-8B 2025.07	28.4	28.4	100
deepseek/deepseek-chat-v3.1 2025.07	28.4	28.4	100
Qwen/Qwen2.5-7B 2025.07	28	28	100
meta-llama/Llama-3.1-8B-Instruct 2025.07	27.5	27.5	100
meta-llama/Llama-3.2-3B 2025.07	23.4	23.4	100
meta-llama/Meta-Llama-3-8B 2025.07	22.6	22.6	100
microsoft/phi-2 2025.07	22.3	22.3	100
meta-llama/Llama-3.1-8B 2025.07	21.8	21.8	100
meta-llama/Llama-3.2-1B 2025.07	21.7	21.7	100
Qwen/Qwen2.5-Math-7B 2025.07	21.4	21.4	100
microsoft/Phi-4-reasoning-plus 2025.07	18.3	18.3	100
microsoft/Phi-3.5-mini-instruct 2025.07	16.9	16.9	100
Qwen/Qwen2.5-72B-Instruct 2025.07	13.4	13.4	100
microsoft/Phi-3-mini-4k-instruct 2025.07	12.8	12.8	100
microsoft/phi-1_5 2025.07	12.5	12.5	100
Qwen/Qwen3-32B 2025.07	9.2	9.2	100
meta-llama/Llama-2-7b-chat-hf 2025.07	8.4	8.4	100
Qwen/Qwen2.5-Math-7B-Instruct 2025.07	7.3	7.3	100
Qwen/Qwen3-14B 2025.07	7.2	7.2	100
microsoft/Phi-4-reasoning 2025.07	4.8	4.8	100
microsoft/phi-1 2025.07	4.4	4.4	100
google/gemini-2.5-pro 2025.07	0.2	0.2	100
deepseek/deepseek-r1-0528 2025.07	0	0	100