Share your thoughts, 1 month free Claude Pro on usSee more

Generative Question Answering on Bolmo Evaluation Suite GenQA 7B

81.6GenQA Average

Llama 3.1 70B

Updated 12d ago

Evaluation Results

Method
Llama 3.1 70B 2025.12	81.6	88.4	91.7	79.6	92.4	78.3	84	53.1	92.9	73.9
Marin 32B 2025.12	80.3	87.2	90.5	76.7	91.1	76.5	80.5	55.1	94.4	70.7
Olmo 3 32B 2025.12	79.8	84.8	90.3	75.7	93.5	80.9	75.3	49	94.5	74.1
OLMo 2 32B 2025.12	79.1	87.5	89.4	77	88.7	76.3	79.1	51.4	94	68.7
Nemo. 3 Nano 2026.04	78.1	86	88.7	75.9	92.5	75	77.2	45.7	94	67.7
Mistral 3.1 24B 2025.12	78	86.2	90.8	79.3	91.9	74.9	80.3	45.1	92.6	61.1
LLM 360 K2V2 70B 2025.12	77.1	87.6	88.9	76.8	90.6	69.7	79.8	47.6	91.2	61.5
Seed 36B 2025.12	76	84.8	89.3	76.1	96	76.1	77.4	30.7	89.1	64.4
Marin 8B 2025.12	75.9	84	88.6	73.9	85.6	73	72.7	42.6	93.4	69.5
Kimi Linear 2026.04	75.7	85	88.2	77.2	93.1	72.2	77.1	25.9	91.8	70.3
Gemma 2 9B 2025.12	75.6	81.8	88.8	76.3	89.3	68.2	75.1	40.4	88.8	71.5
Apertus 70B 2025.12	75	84.5	87.7	74.8	87.5	56.3	77.2	43.1	90.7	72.8
Gemma 3 27B 2025.12	73.5	86	91.3	77.5	94.9	75.9	82.1	49.2	92.4	12.4
Llama 3.1 8B 2025.12	73.1	81.5	87.3	75.5	88	59.5	70.9	36.7	89.2	69
Gemma 2 27B 2025.12	72.9	86.7	90.8	76.9	93.2	73.2	80.7	47.1	93	14.9
Olmo Hybrid 2026.04	72.9	79	86.2	70.2	89.7	72.9	64	34.8	92.1	67.4
Olmo 3 7B 2025.12	72.5	77.7	85.7	68.9	89.5	71.5	60.4	32.6	93.5	72.8
Olmo 3 2026.04	72.5	77.7	85.6	68.8	89.5	71.5	60.3	32.5	93.6	72.8
OLMo 2 7B 2025.12	72.4	82.2	87.4	70.5	82.2	61.5	70.8	37.4	91.5	68.3
Nemo-Nano 9B 2025.12	71.8	80.2	86.2	67.9	91.4	71.4	64.9	31.2	92.3	60.4
Falcon H1 2026.04	71.7	80.9	87.9	73	93.3	69.4	61.1	27.4	91.4	60.7
MiMo 7B 2025.12	71.4	80.6	86.5	73.1	89.7	69.3	65.6	33.1	90.3	54.4
Nemotron-H 2026.04	71.2	84.3	89.6	76	90.2	65.7	69.7	37.7	87.9	39.9
Qwen2 7B 2025.12	71.1	80.5	86.4	73	93.5	57.2	65.1	33.8	89.2	61.6
Qwen 3 2026.04	71.1	80.6	86.4	72.8	93.4	57.2	65.1	33.8	89	61.8
Apertus 8B 2025.12	69	81	85.8	70.9	83.8	37.1	70.1	35	89.6	67.4
Qwen 2.5 32B 2025.12	68.5	86.3	87.5	76.2	94.2	53.7	74	39.3	64.9	40.4
Recurr. Gemma 2026.04	68.5	81.3	86.3	72.2	83	44	66.4	30.6	85.9	66.6
Falcon Mamba 2026.04	68.5	81.9	88.3	80.9	86.1	41.6	67.1	35.1	75.9	59.7
Granite 3.3 8B 2025.12	67.8	83.7	89.4	76	88.7	38.4	69.7	37	89.6	37.8
Qwen 2.5 7B 2025.12	67.5	81	86	70.3	91.4	56.7	63	31.2	87	40.5
Gaperon 24B 2025.12	65.3	75.2	80.3	58.3	83.2	59.4	58.9	33.5	89.3	49.8
Gaperon 8B 2025.12	63.3	73.9	76.4	67	80.5	54.9	55.5	28.8	86	46.7
xLSTM 2026.04	34.8	61.1	62.1	19.7	60.9	24.4	9.3	1.8	33.9	40.4
Olmo 3 7B 2025.12	0.724	0.778	0.857	0.68	0.9	0.715	0.603	0.326	0.935	0.727
TFree-Hat 7B 2025.12	0.713	0.828	0.882	0.705	0.896	0.486	0.683	0.343	0.886	0.711
Bolmo 7B 2025.12	0.709	0.788	0.855	0.711	0.896	0.652	0.568	0.286	0.916	0.705
BLT 7B 2025.12	0.684	0.811	0.882	0.728	0.845	0.388	0.673	0.292	0.852	0.686
EvaByte 6.5B 2025.12	0.414	0.701	0.782	0.629	0.827	0.078	0.131	0.054	0.359	0.167