Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

In harmony with gpt-oss

About

No one has independently reproduced OpenAI's published scores for gpt-oss-20b with tools, because the original paper discloses neither the tools nor the agent harness. We reverse-engineered the model's in-distribution tools: when prompted without tool definitions, gpt-oss still calls tools from its training distribution with high statistical confidence -- a strong prior, not a hallucination. We then built a native harmony agent harness (https://github.com/borislavmavrin/harmonyagent.git) that encodes messages in the model's native format, bypassing the lossy Chat Completions conversion. Together, these yield the first independent reproduction of OpenAI's published scores: 60.4% on SWE Verified HIGH (published 60.7%), 53.3% MEDIUM (53.2%), and 91.7% on AIME25 with tools (90.4%).

Borislav Mavrin• 2026

Related benchmarks

TaskDatasetResultRank
Software EngineeringSWE-Bench--
11
Software EngineeringSWE-bench Verified--
4
Mathematical ReasoningAIME MEDIUM reasoning with tools 2025
Score91.7
2
Software EngineeringSWE Verified MEDIUM reasoning
Overall Score53.3
2
Software EngineeringSWE Verified HIGH reasoning
Accuracy60.4
2
Showing 5 of 5 rows

Other info

Follow for update