MuSR
Benchmarks
| Task Name | Dataset Name | SOTA Result | Trend | |
|---|---|---|---|---|
| Multistep Reasoning | MuSR | Accuracy73.33 | 53 | |
| Reasoning | MuSR 0-shot | Reasoning Score (0-shot)48.82 | 46 | |
| Multistep Soft Reasoning | MuSR | Accuracy69 | 41 | |
| Reasoning | MuSR | Accuracy71.89 | 32 | |
| Multistep Reasoning | MUSR | Accuracy61.67 | 31 | |
| Multistep Soft Reasoning | MUSR | Accuracy (Multi-choice)50.77 | 27 | |
| Math & Logic | MUSR | MUSR Performance42.12 | 24 | |
| Multi-step Narrative Reasoning | MUSR | Accuracy65.86 | 22 | |
| Reasoning | MuSR (test) | Accuracy73.9 | 17 | |
| Multi-step Soft Reasoning | MuSR | Accuracy65.3 | 12 | |
| Murder Mystery Reasoning | MUSR murder (test) | Accuracy75 | 12 | |
| Question Answering | MuSR n=756 (test) | Activation Score98.8 | 11 | |
| Multistep Soft Reasoning | MuSR n=756 (test) | Activation (Act)100 | 11 | |
| Reasoning | MuSR | MuSR Score56.48 | 10 | |
| Multi-hop Reasoning | MuSR | Accuracy43.12 | 10 | |
| Reasoning | MuSR | MuSR Score37.14 | 9 | |
| Self-doubt detection | MuSR 90-trace | AUROC (Self-doubt)83.66 | 7 | |
| Adding Mistake | MuSR | AOC0.731 | 7 | |
| Truncated CoT Answering | MuSR | AOC33.6 | 7 | |
| Multi-step narrative inference | MuSR | Exact Match68.76 | 6 | |
| Multistep Reasoning | MUSR team (test) | Accuracy69 | 6 | |
| Multistep Reasoning | MUSR object (test) | Accuracy68.9 | 6 | |
| Team Allocation Reasoning | MUSR team (test) | Accuracy69 | 6 | |
| Object Placement Reasoning | MUSR object (test) | Accuracy68.9 | 6 | |
| Multistep Reasoning | MUSR-fr | Average Score33.79 | 6 |
Showing 25 of 30 rows