Code Generation on HumanEval 0-shot (TPS, NFE, Acc)
57.93Accuracy
Vanilla
Updated 29d ago
Evaluation Results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
2026.04 | 57.93 | - | 256 | 4.7 | - | - | - | |
2026.04 | 57.93 | - | 80.5 | 1.7 | - | - | - | |
2026.05 | 56.82 | 14.95 | - | - | - | 1 | - | |
2026.05 | 56.78 | 31.99 | - | - | - | 2.14 | - | |
2026.05 | 56.71 | 26.8 | - | - | - | 2.42 | - | |
2026.05 | 56.7 | 3.4 | - | - | 512 | 1 | - | |
2026.05 | 55.7 | 20.18 | - | - | - | 1.35 | - | |
2026.04 | 55.5 | 38.8 | 137 | - | - | - | - | |
2026.05 | 55.5 | 12.7 | - | - | 94 | 4.9 | - | |
2026.05 | 54.93 | 27.81 | - | - | - | 1.86 | - | |
2026.05 | 54.69 | 19.14 | - | - | - | 1.28 | - | |
2026.05 | 54.31 | 24.22 | - | - | - | 1.62 | - | |
2026.05 | 54.3 | 26 | - | - | 65 | 7.5 | - | |
2026.05 | 54.3 | 33.9 | - | - | 50 | 8.8 | - | |
2026.04 | 54.27 | - | 44.6 | 0.9 | - | - | - | |
2026.05 | 54.27 | 12.8 | - | - | - | 1.16 | - | |
2026.05 | 54.27 | 31.37 | - | - | - | 2.83 | - | |
2026.05 | 54.27 | 31.62 | - | - | - | 2.86 | - | |
2026.05 | 54.05 | 36.93 | - | - | - | 2.47 | - | |
2026.04 | 53.7 | 39.4 | 131.8 | - | - | - | - | |
2026.04 | 53.7 | 38.6 | 137.9 | - | - | - | - | |
2026.04 | 53.66 | - | 75 | 1.4 | - | - | - | |
2026.04 | 53.66 | - | 50.4 | 1.1 | - | - | - | |
2026.04 | 53.05 | - | 67.5 | 1.3 | - | - | - | |
2026.04 | 53 | 39 | 135.3 | - | - | - | - | |
2026.05 | 52.44 | 11.07 | - | - | - | 1 | - | |
2026.05 | 52.4 | 12.4 | - | - | 76 | 4 | - | |
2026.05 | 51.83 | 12.28 | - | - | - | 1.11 | - | |
2026.04 | 50.6 | 39.5 | 132.5 | - | - | - | - | |
2026.04 | 50.6 | 39.8 | 127.3 | - | - | - | - | |
2026.04 | 50.6 | 40.6 | 125.3 | - | - | - | - | |
2026.04 | 49.4 | 20.7 | 256 | - | - | - | - | |
2026.06 | 44.51 | 92.52 | - | - | - | - | - | |
2026.05 | 43.9 | 16.98 | - | - | - | 1.05 | - | |
2026.04 | 43.3 | 51 | 86.4 | - | - | - | - | |
2026.04 | 43.3 | 56.5 | 77.5 | - | - | - | - | |
2026.04 | 43.3 | 60.4 | 72 | - | - | - | - | |
2026.05 | 43.29 | 30.18 | - | - | - | 1.87 | - | |
2026.04 | 42.7 | 16.7 | 256 | - | - | - | - | |
2026.04 | 42.7 | 53.4 | 80.6 | - | - | - | - | |
2026.04 | 42.7 | 59.8 | 74.8 | - | - | - | - | |
2026.05 | 42.7 | 20.1 | - | - | 219 | 1.9 | - | |
2026.05 | 42.7 | 36.5 | - | - | 125 | 3.3 | - | |
2026.05 | 42.68 | 5.77 | - | - | - | 1 | - | |
2026.05 | 42.68 | 15 | - | - | - | 2.6 | - | |
2026.05 | 42.68 | 5.46 | - | - | - | 0.95 | - | |
2026.05 | 42.68 | 20.71 | - | - | - | 3.59 | - | |
2026.05 | 42.68 | 33.68 | - | - | - | 2.09 | - | |
2026.05 | 42.3 | 10.8 | - | - | 512 | 1 | - | |
2026.04 | 42.1 | 16.1 | 93.4 | - | - | - | - | |
2026.04 | 42.1 | 16.4 | 93 | - | - | - | - | |
2026.05 | 42.1 | 30.2 | - | - | 176 | 2.8 | - | |
2026.06 | 41.7 | 70.3 | 20,463 | - | - | - | 3.75 | |
2026.04 | 41.5 | 57.2 | 76.4 | - | - | - | - | |
2026.04 | 41.5 | 5.9 | 256 | - | - | - | - | |
2026.04 | 41.5 | 16.2 | 98.5 | - | - | - | - | |
2026.06 | 41.5 | 54.1 | 27,366 | - | - | - | 2.8 | |
2026.06 | 41.5 | 75.5 | 18,909 | - | - | - | 4.05 | |
2026.05 | 41.46 | 55.11 | - | - | - | 3.31 | - | |
2026.05 | 41.46 | 10.83 | - | - | - | 1.88 | - | |
2026.05 | 41.46 | 19.15 | - | - | - | 3.32 | - | |
2026.04 | 40.9 | 50.4 | 86.2 | - | - | - | - | |
2026.04 | 40.9 | 14.2 | 106.9 | - | - | - | - | |
2026.06 | 40.9 | 107.7 | 9,740 | - | - | - | 4.06 | |
2026.06 | 40.7 | 89.6 | 10,538 | - | - | - | 3.78 | |
2026.04 | 40.24 | - | 256 | 5.8 | - | - | - | |
2026.05 | 40.24 | 16.66 | - | - | - | 1 | - | |
2026.05 | 40.24 | 31.81 | - | - | - | 1.91 | - | |
2026.05 | 40.24 | 67.99 | - | - | - | 4.08 | - | |
2026.05 | 40.24 | 70.65 | - | - | - | 4.24 | - | |
2026.06 | 40.24 | 71.72 | - | - | - | - | - | |
2026.04 | 40.2 | 15.3 | 99.6 | - | - | - | - | |
2026.06 | 40.2 | 77.8 | 13,666 | - | - | - | 2.87 | |
2026.05 | 39.63 | 32.51 | - | - | - | 1.95 | - | |
2026.05 | 39.63 | 24.73 | - | - | - | 1.54 | - | |
2026.05 | 39.63 | 33.31 | - | - | - | 2.07 | - | |
2026.04 | 39.6 | 14.8 | 105.6 | - | - | - | - | |
2026.05 | 39.02 | 16.1 | - | - | - | 1 | - | |
2026.06 | 39.02 | 112.61 | - | - | - | - | - | |
2026.06 | 38.47 | 140.58 | - | - | - | - | - | |
2026.06 | 38.22 | 138.58 | - | - | - | - | - | |
2026.04 | 37.8 | 14.1 | 110.9 | - | - | - | - | |
2026.06 | 37.8 | 53.7 | 25,465 | - | - | - | 1.64 | |
2026.06 | 37.8 | 58.7 | 44,862 | - | - | - | 1.87 | |
2026.06 | 37.8 | 60.84 | - | - | - | - | - | |
2026.06 | 37.8 | 137.88 | - | - | - | - | - | |
2026.06 | 37.46 | 35.88 | - | - | - | - | - | |
2026.06 | 37.37 | 11.28 | - | - | - | - | - | |
2026.06 | 37.2 | 72.7 | 37,168 | - | - | - | 2.25 | |
2026.06 | 37.19 | 13.52 | - | - | - | - | - | |
2026.06 | 36.6 | 71 | 37,740 | - | - | - | 2.22 | |
2026.04 | 36.59 | - | 90.1 | 2.5 | - | - | - | |
2026.04 | 36.59 | - | 64.3 | 1.7 | - | - | - | |
2026.06 | 36.01 | 1.19 | - | - | - | - | - | |
2026.06 | 36 | 68.3 | 20,246 | - | - | - | 2 | |
2026.04 | 35.98 | - | 69.8 | 2.1 | - | - | - | |
2026.04 | 35.98 | - | 81.7 | 2.3 | - | - | - | |
2026.06 | 35.98 | 6.04 | - | - | - | - | - | |
2026.06 | 35.4 | 70.6 | 20,156 | - | - | - | 2.14 | |
2026.04 | 33.53 | - | 37.9 | 1.1 | - | - | - |