Spider
Benchmarks
| Task Name | Dataset Name | SOTA Result | Trend | |
|---|---|---|---|---|
| Text-to-SQL | Spider (test) | Execution Accuracy91.2 | 256 | |
| Text-to-SQL | Spider (dev) | EX90.3 | 196 | |
| Text-to-SQL | Spider | Exec Acc (All)98.65 | 139 | |
| Text-to-SQL | Spider-DK | Execution Accuracy (EX)81.9 | 136 | |
| Text-to-SQL | Spider-Syn | Execution Accuracy (EX)83.1 | 120 | |
| Text-to-SQL | Spider 1.0 (test) | EM Acc (Overall)91.2 | 110 | |
| Text-to-SQL | Spider 1.0 (dev) | Exact Match Accuracy86.75 | 92 | |
| Text-to-SQL | Spider Realistic | Execution Accuracy (EX)88.6 | 55 | |
| Dataset-level accuracy estimation | Spider to SynSQL 2.5M | MAE2.8 | 54 | |
| Dataset-level accuracy estimation | Spider to BIRD | MAE3.1 | 54 | |
| Text-to-SQL | Spider-Realistic | Execution Accuracy (EX)88.6 | 50 | |
| Text-to-SQL | Spider Lite 2.0 | Execution Accuracy (EX)69.3 | 46 | |
| Text-to-SQL | Spider | Attack Accuracy41.73 | 40 | |
| Text-to-SQL | Spider | Execution Accuracy (EX)88.4 | 38 | |
| Text2SQL | Spider (test) | Exec Acc (Greedy)88.3 | 37 | |
| Text-to-SQL | Spider Non-Synthesized Matched Set | Execution Match Accuracy (ExM)97.31 | 32 | |
| Text-to-SQL Error Repairing | Spider | True Positive Repairs (EM)116 | 30 | |
| Text-to-SQL | Spider Non-Seed Domains | Non-Seed Domain Score64.7 | 30 | |
| Text-to-SQL | Spider all domains | SemanticScore58.3 | 30 | |
| text-to-SQL | Spider | Accuracy82.5 | 28 | |
| Text-to-SQL | Spider 2.0 (test) | Execution Accuracy (Spider 2.0 Test)26.7 | 27 | |
| SQL Semantic Validation | Spider | AUPRC63.21 | 24 | |
| Text-to-SQL | Spider | AVGLEN1.07 | 22 | |
| Text-to-SQL | Spider 2.0 | Llama-8B Execution Accuracy4.4 | 21 | |
| Text-to-SQL | Spider Snow 2.0 (test) | Execution Accuracy (EX)60.51 | 21 |
Showing 25 of 111 rows