Spider
Benchmarks
| Task Name | Dataset Name | SOTA Result | Trend | |
|---|---|---|---|---|
| Text-to-SQL | Spider (test) | Execution Accuracy89.65 | 162 | |
| Text-to-SQL | Spider (dev) | EX (All)89.5 | 100 | |
| Text-to-SQL | Spider 1.0 (dev) | Exact Match Accuracy86.75 | 92 | |
| Text-to-SQL | Spider | Exec Acc (All)92.2 | 91 | |
| Text-to-SQL | Spider 1.0 (test) | EM Acc (Overall)87.2 | 91 | |
| Dataset-level accuracy estimation | Spider to SynSQL 2.5M | MAE2.8 | 54 | |
| Dataset-level accuracy estimation | Spider to BIRD | MAE3.1 | 54 | |
| Text2SQL | Spider (test) | Exec Acc (Greedy)88.3 | 37 | |
| Text-to-SQL | Spider-Realistic | Execution Accuracy (EX)81.9 | 33 | |
| Text-to-SQL | Spider-Syn | Gre78.6 | 32 | |
| Text-to-SQL | Spider-DK | Gre Score76.8 | 32 | |
| Text-to-SQL | Spider Lite 2.0 | Execution Accuracy (EX)49.18 | 24 | |
| SQL Semantic Validation | Spider | AUPRC63.21 | 24 | |
| Text-to-SQL | Spider | AVGLEN1.07 | 22 | |
| Schema retrieval | Spider | Recall100 | 19 | |
| Semantic Validation | Spider 2.0 | AUPRC92.59 | 18 | |
| SQL execution performance | Spider n=1034 | EM (1 Table)84.3 | 18 | |
| Lossless Compression | Spider (test) | bits/Byte0.723 | 12 | |
| Text-to-SQL | Spider 1.0 (val) | Accuracy (All)64.8 | 11 | |
| Semantic Parsing | Spider (dev) | Exact Match Accuracy75.5 | 11 | |
| Text-to-SQL | Spider-Syn (dev) | Exact Match Accuracy62.6 | 11 | |
| Table Selection | SPIDER 2018 (test) | Avg Tables2.7 | 10 | |
| Text-to-SQL | Spider hidden (test) | Exact Match (EM)72.1 | 10 | |
| Text-to-SQL | Spider-SYN 1.0 (val) | EM Accuracy66.9 | 10 | |
| Text-to-SQL | Spider ADVETA-ADD | Exact Match (EM)50.6 | 10 |
Showing 25 of 64 rows