tau2-bench
Benchmarks
| Task Name | Dataset Name | SOTA Result | Trend | |
|---|---|---|---|---|
| Interactive Tool-Use Agent Performance | tau2-Bench | Retail Performance Score81.1 | 102 | |
| Agentic Tool-use | tau2-bench Airline | Pass@163.5 | 30 | |
| Agentic Tool-use | tau2-bench Retail | Pass@182 | 30 | |
| Multi-turn Decision Making | tau2-Bench held-out (test) | Average Score41.9 | 16 | |
| Task failure prediction and selective task completion | tau2-bench Telecom 1.0 | AUROC0.809 | 15 | |
| Task failure prediction and selective task completion | tau2-bench Retail 1.0 | AUROC0.707 | 15 | |
| Task failure prediction and selective task completion | tau2-bench Airline 1.0 | AUROC74.2 | 15 | |
| Stateful Interaction | tau2-bench | Score86.69 | 12 | |
| Long-Horizon Tool Execution | tau2-Bench | Retail Success Rate75.2 | 12 | |
| Real-world issue resolution | tau2-bench | Airline Success Rate72 | 9 | |
| Agent Task Completion | tau2-bench telecom | Pass Rate67 | 9 | |
| Agent Task Completion | tau2-bench airline | Pass Rate60 | 9 | |
| Agent Task Success | tau2-bench Retail Domain | Total Pass Rate61.4 | 9 | |
| Agentic Skill Acquisition | tau2-bench | Pass@181.2 | 9 | |
| Tool-use Task Completion | Tau2-Bench Telecom subset Solo mode | Success Rate43.3 | 7 | |
| Multi-turn agent decision making | tau2-Bench (test) | Success Rate22.3 | 7 | |
| False Success Detection | tau2-bench v1 v2 (test) | Recall89 | 6 | |
| General Task (Agentic Coding) | tau2-Bench Telecom | Score98.2 | 6 | |
| Tool Use | Tau2-Bench | Success Rate57.4 | 6 | |
| User Simulation Behavioral Alignment | tau2-bench Retail + Airline (test) | HL Score95.8 | 5 | |
| User Simulation Behavioral Alignment | tau2-bench Airline (test) | HL90.3 | 5 | |
| Failure prediction | tau2-bench Transfer held-out | ROC-AUC67.4 | 4 | |
| Failure prediction | tau2-bench Within corpus | ROC AUC0.719 | 4 | |
| Tool Use | tau2-bench | Productivity43 | 4 | |
| Stateful service dialogue | tau2-bench Telecom | Task Completion Score (TCS)85.1 | 4 |
Showing 25 of 30 rows