COUNTDOWN
Benchmarks
| Task Name | Dataset Name | SOTA Result | Trend | |
|---|---|---|---|---|
| Mathematical Reasoning | Countdown | Accuracy85 | 252 | |
| Planning | Countdown | Accuracy87.5 | 89 | |
| Mathematical Reasoning | Countdown (test) | Accuracy85 | 84 | |
| Reasoning | Countdown | Accuracy83.2 | 49 | |
| Planning | Countdown | Accuracy85.6 | 27 | |
| Logical planning | Countdown (test) | Accuracy84.8 | 24 | |
| Symbolic Reasoning | Countdown | Accuracy49.61 | 24 | |
| Arithmetic Reasoning | Countdown | Accuracy33.6 | 19 | |
| Logical Reasoning | Countdown | Accuracy52 | 16 | |
| Arithmetic Reasoning | Countdown 512 tokens | Pass@162.1 | 15 | |
| Arithmetic Reasoning | Countdown 256 tokens | Pass@171.1 | 15 | |
| Planning and Reasoning | Countdown | Accuracy81.6 | 14 | |
| Planning | Countdown (held-out) | Pass@187.96 | 14 | |
| Logical Reasoning | Countdown CD34 | Avg@1678.2 | 14 | |
| Logical Reasoning | Countdown CD4 | Avg@1659.4 | 14 | |
| Numerical Reasoning | Countdown-4 | CD498.9 | 13 | |
| Mathematical Reasoning | Countdown | Accuracy (L=128)54.3 | 13 | |
| Reasoning | COUNTDOWN (test) | Accuracy66.02 | 13 | |
| Countdown | Countdown 3 (test) | Reward95 | 12 | |
| Mathematical Reasoning | Countdown 4,5,6-arg held-out difficulties (test) | Accuracy25.1 | 10 | |
| Mathematical Reasoning | Countdown 8B Instruct (test) | Accuracy46.1 | 9 | |
| Mathematical Reasoning | Countdown-34 (held-out) | Accuracy81.26 | 8 | |
| Uncertainty Quantification | Countdown | ROC-AUC (128)0.61 | 8 | |
| Arithmetic Reasoning | Countdown 0-shot (test) | Pass@1 (Greedy)71.5 | 7 | |
| Mathematical Reasoning | Countdown | CD4 Score49.8 | 6 |
Showing 25 of 35 rows