Sequential environment decision making on Crafter BALROG protocol

57.3Peak Task Score (%)

Gemini-3-Pro

Updated 23d ago

Evaluation Results

Method
Gemini-3-Pro 2026.07	57.3	-
Gemini-3.1-Pro-Thinking 2026.07	55	-
Gemini-2.5-Pro 2026.07	55	-
AUTOMEM 2026.07	51.36	-
Claude-Opus-4.5 2026.07	49.5	-
AUTOMEM 2026.07	47.27	-
MAGE 2026.05	37.9	-
MAGE Opus–Llama (n=5, peak) 2026.05	37.9	4
DeepSeek-R1 2026.07	36.4	-
MAGE 2026.05	34.8	-
MAGE Haiku–Llama (n=5, peak) 2026.05	34.8	4
MAGE 2026.05	33.5	-
MAGE Sonnet–Llama (n=4, peak) 2026.05	33.5	4
Qwen2.5-72B-Instruct 2026.07	27.3	-
Standalone Llama-3.1-8B 2026.05	25.5	-
Standalone Llama-3.1-8B (BALROG ref.) 2026.05	25.5	-
AUTOMEM 2026.07	25	-
Qwen2.5-32B-Instruct 2026.07	19.55	-
Qwen2.5-32B-Instruct 2026.07	17.27	-
Qwen2.5-7B-Instruct 2026.07	16.4	-