Who & When
Benchmarks
| Task Name | Dataset Name | SOTA Result | Trend | |
|---|---|---|---|---|
| Error Attribution | Who&When | Pair µF18.1 | 30 | |
| Failure Attribution | Who&When | Agent Accuracy60.79 | 22 | |
| Error Recognition | WHO&WHEN Algorithm-Generated | Accuracy@038.9 | 19 | |
| Error Recognition | WHO&WHEN Human-Crafted | Acc@017.2 | 19 | |
| Fault attribution | Who&When Hand (58 traces) | Top Accuracy31 | 16 | |
| Fault attribution | Who&When Algo (126 traces) | Accuracy49.2 | 16 | |
| Failure Attribution | Who&When (HC) | Accuracy (%)22.99 | 15 | |
| Failure Attribution | Who&When (Alg) | Accuracy (Who&When Alg)42.86 | 15 | |
| Trajectory Attribution | Who&When n=58 (Hand-Crafted) | Agent-level Accuracy73 | 15 | |
| Trajectory Attribution | Who&When Algorithm-Generated n=126 | Agent-level Accuracy68 | 15 | |
| Failure Attribution | Who&When Total | Step-level Accuracy36.22 | 13 | |
| Failure Attribution | Who&When Hand-Crafted | Step-level Accuracy41.38 | 13 | |
| Failure Attribution | Who&When Algorithm-Generated | Step-level Accuracy42.86 | 13 | |
| Online auditing | Who&When | Step Accuracy57.69 | 8 | |
| Error Forecasting | Who&When | Eta (%)100 | 6 | |
| Failure attribution | Who & When Boundary | Agent Attribution Accuracy38.71 | 6 | |
| Failure attribution | Who & When Remove ID | Agent Attribution Accuracy26.47 | 6 | |
| Failure attribution | Who & When Baseline | Agent Attribution Accuracy54.33 | 6 | |
| Step-level fault attribution | Who&When HC | Top-1 Accuracy21.26 | 5 | |
| Step-level fault attribution | Who&When (Alg) | Top-1 Accuracy28.84 | 5 | |
| Step-level attribution | Who&When attribution benchmark | Rec-Exact60 | 4 | |
| Step-level failure attribution | Who&When HC | Input Tokens79,817.71 | 4 | |
| Step-level failure attribution | Who&When (Alg) | Input Tokens15,673.79 | 4 | |
| Fault localization | Who&When | Top-1 Accuracy21.1 | 3 |
Showing 24 of 24 rows