Share your thoughts, 1 month free Claude Pro on usSee more

Long-context Question Answering on LongBench V2

56.77Overall Accuracy

Gemini-2.5-Flash-Thinking

Updated 26d ago

Evaluation Results

Method
Gemini-2.5-Flash-Thinking 2026.01	56.77	51.43	55.2	58	66.67	-	-	-	-	-	-	-	-	-	-
LONGPAS 2026.01	54.27	47.43	54.4	60	76.92	-	-	-	-	-	-	-	-	-	-
LONGPAS 2026.01	49.11	46.29	42.4	56	71.79	-	-	-	-	-	-	-	-	-	-
Qwen3-30B-A3B-Thinking 2026.01	48.31	44	44	46	64.1	-	-	-	-	-	-	-	-	-	-
RLVR 2026.01	47.91	50.29	50.4	42	56.41	-	-	-	-	-	-	-	-	-	-
GPT-OSS-120B 2026.01	47.01	44.57	43.2	53.06	61.54	-	-	-	-	-	-	-	-	-	-
RLVR 2026.01	44.53	40	46.4	50	51.28	-	-	-	-	-	-	-	-	-	-
Qwen3-30B-A3B-Instruct 2026.01	44.43	42.43	38.4	52	62.82	-	-	-	-	-	-	-	-	-	-
GPT5-Nano 2026.01	43.74	44	39.2	50	46.15	-	-	-	-	-	-	-	-	-	-
GPT-OSS-20B 2026.01	43.37	38.51	40.8	56	61.54	-	-	-	-	-	-	-	-	-	-
LONGPAS 2026.01	42.94	38.14	40.6	48	63.46	-	-	-	-	-	-	-	-	-	-
MemSearch-o1 2026.04	42.31	-	-	-	-	43.48	40.43	42.42	48.65	36.59	-	-	-	-	-
LONGPAS 2026.01	42.3	40.29	40	43	62.82	-	-	-	-	-	-	-	-	-	-
RLVR 2026.01	42.1	39.14	37	49.5	60.9	-	-	-	-	-	-	-	-	-	-
RLVR 2026.01	41.75	37.14	40.8	46	66.67	-	-	-	-	-	-	-	-	-	-
Qwen3-4B-Thinking 2026.01	40.46	37	35.6	41.5	60.26	-	-	-	-	-	-	-	-	-	-
A-Mem 2026.04	39.33	-	-	-	-	39.13	40.43	42.42	40.54	34.15	-	-	-	-	-
Qwen3-4B-Instruct 2026.01	37.28	36	30.6	36.5	60.26	-	-	-	-	-	-	-	-	-	-
Amber 2026.04	35.85	-	-	-	-	39.13	39.36	36.36	35.14	29.27	-	-	-	-	-
LONGPAS 2026.01	33.7	39.57	28.4	29.5	31.41	-	-	-	-	-	-	-	-	-	-
Qwen2.5-7B-Instruct 2026.01	33.6	36.71	28.2	32	35.9	-	-	-	-	-	-	-	-	-	-
KV-CAT 2026.05	31.2	-	-	-	-	-	26.6	42.4	-	-	30	26.7	33.3	35	33.3
RLVR 2026.01	31.01	34.43	29	30	40.38	-	-	-	-	-	-	-	-	-	-
Search-o1 (Refined) 2026.04	30.96	-	-	-	-	30.43	27.27	32.98	32.43	31.71	-	-	-	-	-
KV-CAT 2026.05	30.8	-	-	-	-	-	27.7	45.5	-	-	30	26.7	28.6	35	22.2
KV-CAT 2026.05	30.3	-	-	-	-	-	26.6	48.5	-	-	30	33.3	19	30	27.8
LONGPAS 2026.01	29.62	30.14	29.8	32	34.62	-	-	-	-	-	-	-	-	-	-
RLVR 2026.01	28.93	31.86	29.2	29	32.69	-	-	-	-	-	-	-	-	-	-
Direct RAG 2026.04	28.91	-	-	-	-	30.43	33.33	24.24	29.73	26.83	-	-	-	-	-
LLaMA3.1-8B-Instruct 2026.01	27.93	28.86	26.2	27	32.05	-	-	-	-	-	-	-	-	-	-
Base 2026.05	25.3	-	-	-	-	-	21.3	27.3	-	-	30	20	38.1	15	38.9
Base 2026.05	22.2	-	-	-	-	-	18.1	27.3	-	-	20	33.3	28.6	30	11.1
Base 2026.05	22.2	-	-	-	-	-	19.1	27.3	-	-	25	26.7	28.6	15	22.2