In harmony with gpt-oss
About
No one has independently reproduced OpenAI's published scores for gpt-oss-20b with tools, because the original paper discloses neither the tools nor the agent harness. We reverse-engineered the model's in-distribution tools: when prompted without tool definitions, gpt-oss still calls tools from its training distribution with high statistical confidence -- a strong prior, not a hallucination. We then built a native harmony agent harness (https://github.com/borislavmavrin/harmonyagent.git) that encodes messages in the model's native format, bypassing the lossy Chat Completions conversion. Together, these yield the first independent reproduction of OpenAI's published scores: 60.4% on SWE Verified HIGH (published 60.7%), 53.3% MEDIUM (53.2%), and 91.7% on AIME25 with tools (90.4%).
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Software Engineering | SWE-Bench | -- | 11 | |
| Software Engineering | SWE-bench Verified | -- | 4 | |
| Mathematical Reasoning | AIME MEDIUM reasoning with tools 2025 | Score91.7 | 2 | |
| Software Engineering | SWE Verified MEDIUM reasoning | Overall Score53.3 | 2 | |
| Software Engineering | SWE Verified HIGH reasoning | Accuracy60.4 | 2 |